I grew up in Northern Ireland in the 1980s during the Troubles. For my parents, and just about every other adult, daily life was punctuated with news of bomb attacks, incendiary devices, kidnappings, kneecappings and murders, all motivated in one way or another by a border.

Plenty of children’s lives were blighted, too, but I had no direct experience of the conflict – or even, initially, of the border. Then, one day when I was six, I ended up in a car heading south with my little brother, my parents and my maternal grandparents.

We often went on outings as a family but this one felt different. There were constant mumblings among the adults about how long the journey would be and which route to take. I also recall out-of-character nervous laughter about things that might go wrong. Back then, Protestants like us tended not to cross the border much.

My grandfather had bought a new roadmap and I remember demanding to be map-reader. I had never really looked at a map of the island of Ireland before and I had not, at least not in memory, visited the Republic. Both on paper and in reality, this was my first encounter with a border.

We set off from home, in the countryside near Belfast, in our crowded car and drove for about an hour. I can still hear mum telling me and my brother to be quiet as we entered a slow-moving queue of vehicles running up to a checkpoint. My granny – usually indulgent – supported mum, saying our dad needed to concentrate.

At the checkpoint, the driver of each car was required to wind down their window and answer questions from a member of the British security forces.

“Where are you going today, sir?”

“Is it your own vehicle, sir?”

“Would you have some form of identification on you, sir?”

I would realise in later years that the patter was unchanging and usually delivered with the same thin veneer of politeness. But even then I sensed how the adults in my life bristled. I noticed dad’s staccato replies.

Soon we were off again, driving in an unfamiliar land. I remember my instant fascination with never-before-seen road signs, the different feel and sound of new road surfaces and unusual makes of car.

The adults made comparisons. Was this or that thing better or worse than in Northern Ireland? (Mainly worse, it seemed.) Was this or that thing better or worse than the last time they had been down here? (Mainly unchanged.) There were other conversations too, not really understood, about politics, terrorism and the sympathies and views of people in this border area.

What was the trip for? To my young mind that was the most perplexing things of all. We were heading to a place where my granny’s family had lived and died many decades before. I simply could not understand this. Why had they ended up in this foreign country? What had driven them here? How had they got on with the local people? How had they got by in this land where everything – roads, fields, mountains, houses, postboxes – was alien?

Our destination was the Presbyterian churchyard at Mullafarry, County Mayo, where these relatives had been buried. It was deserted when we saw it, the church still functioning but barely. The gravestone was easily located: Ben and Ellen Ireland (yes, that was their surname), my granny’s great-grandparents. Ben had been a gamekeeper on a local estate and they had lived in Mayo from the 1890s – when there was no border – until long after Ireland’s division in the 1920s.

Afterwards, the day trip acquired a kind of fairytale quality. Whenever it comes to mind, I first see our drive to Mayo as if in drone footage. Simultaneously, I see my ancestors – good people in old-time clothing – moving around the same countryside, living in their farmhouses, helping each other in the fields, walking to church on Sundays, gathered by firesides to play music.

But these idyllic scenes are underpinned by the dramatic tension of the border, because it was the border that struck me most forcefully that day. As with any six-year-old, what I had taken from the trip were just snippets and fragments but they were no less vivid for that. The border as a place of control and interrogation. The border that had made the adults in my life worried and momentarily vulnerable. The people living on the other side of the line: a kind of parallel universe. The mix of relief and regret at getting back to our side that evening.

More than once in the years since, I have been the person enduring the guard’s stare. It happens fairly often to people who, like me, spend lots of time in the former USSR. I have had my bags turned out on Russia’s border with Finland and have been made to delete photographs from a camera near Georgia. I have been accused of having the wrong stamp in my passport more than once and have had to plead – and even to bribe – my way through.

When I approach a frontier, I think back to the Mayo trip and to granny’s instruction to be quiet as dad navigated us through. Really, this was a desire on her part that we would draw no special attention to ourselves, a hope that we would be lucky enough, on this occasion, to slip through.

• The Curtain and the Wall by Timothy Phillips is published by Granta (£20). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply.

• Do you have an opinion on the issues raised in this article? If you would like to submit a letter of up to 300 words to be considered for publication, email it to us at guardian.letters@theguardian.com