I grew up a dark-skinned child in New Orleans in the 1980s when both racism and colourism were rampant. My mother was a light-skinned Black woman, and older girls at my all-Black Catholic school would ask me if I was adopted. When I said no, they would ask how then my mother could be so pretty and I could be so ugly. I was called ugly so often I settled into it; it became not so much a belief as an identity.

When I was 12, my mom and I moved to Connecticut, to a little town called Bethel where we were one of a handful of Black families. The white kids at school threw the N-word around liberally. By contrast, my mother was Ms Black Pride. She drove 30 miles out of her way to buy doughnuts from the Black-owned bakery; she introduced me to Alex Haley’s Roots when I was seven years old. When Spike Lee came to New Orleans to screen Malcolm X, we went, then when the movie was released, we saw it three more times in the theatre. When I was in fifth grade, she pulled me out of my integrated school and enrolled me in an all-Black one. I pleaded with her to let me stay as I had some friends and familiarity there, but she refused. “They don’t see you as equal,” she said. Years later, I understood that she was right.

High school was better. It was still predominantly white, but “the minorities” – as we called ourselves – established our own group to feel safe. Dark skin wasn’t as much of an offence in Connecticut; and, albeit slowly, darker-skinned women were being cast in television shows and music videos. Lauryn Hill had just released The Miseducation of Lauryn Hill. I had matured, gained a certain ease in myself. I went into college balancing the Black pride that had been planted by my mother, my more positive high school experiences and the wounds I had accumulated.

I met my husband in my third year of college. We went to Dartmouth College in New Hampshire, and ended up choosing the same foreign study programme in Trinidad. We would learn later that we’d had two classes together on campus, but we’d never spoken. There was a group of eight of us from Dartmouth on the trip, and we all lived in The University of the West Indies (UWI) at St Augustine’s dormitory. Though we were in a country comprised primarily of people of colour, and though I’d assumed upon entering that I’d immediately meet my Trinidadian husband, our curriculum necessitated that the eight of us spend all our time together. For the first time in a long time, despite being in a country where whites were the minority, I was the only Black person.

I can’t tell you how uncomfortable it made me. I was constantly on edge for any inappropriate comment, any slur, any indication that my white peers had internalised the hierarchy inherent in white supremacy. I felt out of control of my own worth. That feeling never quite left me, but after nights being hazed by the local UWI students and weeks of drunken shenanigans at Coconuts, a club in Port of Spain, I began to relax. I bonded with one of the three white guys and, after a few weeks, we learned that we both loved Tracy Chapman, and that his father and I had gone to the same school. We were both known in the group for our laughter and, in discussions about Salman Rushdie’s Midnight’s Children or Buchi Emecheta’s The Joys of Motherhood, we were drawn to the same side of the analysis. One night, there was a particularly long line at one of our favourite clubs. I had gone on the first bus and arrived earlier than him. While I waited, I bought his favourite, rum and Coke, and as soon as he walked in, I held it out to him. When we finished our drinks, he led me to the dancefloor. Tanya Stephens’ It’s a Pity was playing. I slipped my hand in his pocket, and all our other plans got cancelled.

* * *

We still talk about that night with nostalgia: the rum, the dancefloor, the youth. But if I was on edge before we started dating, I became chronically anxious after. I had a distinct fear that someone was going to discriminate against me in front of him and he would then view me the way society stereotypically views Black people. (Again, when my children were born, I feared that some white person would denigrate me in front of them, and I’d lose a sense of my own authority, the confidence I feel as their protector and guide.) If that were to happen, I knew he would stand beside me, help to defend me, but I worried that our power dynamic would be irretrievably altered because we would both know that of the two of us, one was viewed as more worthwhile. I worried I wouldn’t be able to shake off how small that might make me feel. My boyfriend, on the other hand, was in the grips of young love; he didn’t think about the racial dynamics at all.

We never did knowingly encounter discrimination in Trinidad, nor when we got back to college and spent the next year-and-a-half with our combined groups of friends – my tribe of Black women and his freshman year roommates, each a distinct ethnicity. It didn’t happen when we lived in New York the summer after graduation, either. But we travelled to the Dominican Republic in the fall of 2004, and the thing I had feared came upon me.

I had a grant to work at a non-profit organisation that fought the racist laws targeting Dominicans of Haitian descent, who are typically darker than other Dominicans and, on the basis of their skin colour and origin, have been denied basic rights typically associated with citizenship. I remember reading about the issues before I arrived and feeling an instant connection with the victimised demographic. I wasn’t completely healed, but I had come a long way and was excited to see how much distance I’d covered from the girl who prayed every night for her colour to turn. I was excited to see my progress effect real, concrete change.

I entered the country with optimism, but soon had a rude awakening. People often assumed I was my boyfriend’s prostitute; they called him sir and looked at me with disdain; passengers on the local guaguas (buses) would scream “Negra” out the window. I took it as a compliment at first, then a Dominican friend sweetly informed me that it wasn’t intended to be positive. My husband would respond with anger, but he’d also try to placate me, assure me that their ignorant opinions were a product of colonialism. He’d warn me that I’d only lower myself by paying attention.

And just as I feared, despite his anger and his affirmations, my confidence began to wane and our relationship started to change. I became jealous, paranoid that my boyfriend was going to begin to agree with the world I had inhabited. There were white girls who worked with him at his mobile clinic in Cojobal; there were thin light-skinned girls who looked just like the ones who teased me in New Orleans. I’d interrogate him about whether or not he was checking them out, whether or not he would prefer to be with them. I created arguments out of nothing, felt rejected, so I rejected him “back”. Suddenly I was filled with anger. Yes, the anger was targeted at Dominican racism, colourist New Orleanians, white supremacy in general, but it was spilling out on to my boyfriend.

Things settled when we got back to the States, but the old familiar feeling of valuelessness that I had internalised had been dug up by my experience in the Dominican Republic. I pulled back a bit from my boyfriend. On the surface, and even fairly deep beneath it, we were close, but I never let myself be completely vulnerable. It wasn’t that I was afraid he would hurt me directly, it was that I was afraid white supremacy would swoop in again and reduce me. I’d be levelled if I weren’t prepared for it. Moments that should have been joyful – like waiting on an engagement ring when we’d discussed marriage; buying a new home with the prospect of racist lenders – turned gruelling. Some subtly racist act would occur – a white girl in our social circle explaining that another friend had high standards because he was only attracted to blond girls, for instance – and my husband wouldn’t notice it. Or, in the name of soothing me, he’d attribute benign intentions to the offender, and a gulf would spring up between us. But we’d talk, and all my anger would be drilled down to fear, and then to sadness.

We got married and became parents, which cemented our bond. We were our own family now. At this point, society’s judgment of one of us would be a judgment against us all. Language evolved – there began to be words like “microaggression”. I remember when I was a middle-school student and the white kids made fun of me for my New Orleans accent. They’d speak to me in a warped, stereotypically Black manner. One day I worked up the nerve to talk to the vice principal about it. I told her what they were doing and I told her that it was racist. She asked me how it was racist if it was only about the way I spoke? What did that have to do with my colour? I was 11 then, and I didn’t have the words to explain the correlation. I started to cry. Before I left her office, she scolded me for applying such an egregious description to a harmless schoolyard taunt.

I began demanding more from my husband. He may have had positive intentions in drawing my attention away from racial offences, from softening the intent of their source, but his reassurances were naive at best and harmful at worst. They implied that he didn’t feel the slights viscerally, as deeply as though they were his own; if he had, he wouldn’t have been able to miss one when it landed.

He learned well. I got older. I left my job as a lawyer to begin an amorphous writing career, and it actually worked out. I got published, and people actually wanted to read my work. I went to therapy. I became a more seasoned mom. I began to care less and less what people thought of me. In my 20s and even 30s I would have considered someone else’s opinion about me to be impactful, but it became easier to shrug things off, to not let them kill my vibe. Racism had always been the exception to that, but why did it have to be? It reminds me of that Toni Morrison quotation, “If I take your race away, and there you are, all strung out. And all you got is your little self, and what is that? What are you without racism? Are you any good? Are you still strong? Still smart? Do you still like yourself? If you can only be tall because somebody is on their knees, then you have a serious problem.” And that’s a problem that, on an emotional, spiritual and marital level, has nothing to do with me.

My husband now works in diversity, equality and inclusion at a tech company; he leads the white anti-racist group at our children’s school, not because I’ve forced him, but because he has tried to become as ingrained in the cause as he would be if it directly affected him. He is always challenging himself to do more, and it is harder and harder to confuse him with the system. It’s still not perfect and never will be. But recently, a white reporter interviewed us about our relationship separately. Some of the questions were the same, and afterward my husband and I compared notes. She’d said something racist, he thought, something about the burden of being in an interracial relationship being harder on my husband because he was white. He’d corrected her, but he still had feelings about it. I remembered that she had implied something similar in my own interview. It wasn’t that I hadn’t noticed it at the time, but I’d asserted the actual truth and moved on. No part of it had lingered with me. There was no space for it between my husband and me.

• Margaret Wilkerson Sexton is the author of On the Rooftop, published by Oneworld at £9.99. To support the Guardian and Observer, buy your copy at guardianbookshop.com. Delivery charges may apply.