The metallic blue Transit with a horseshoe on the radiator refuses to budge. Damian Le Bas turns the key again. And again. After the eighth attempt, the engine rattles into life. “Thank fuck,” he exhales.

Le Bas, 33, a writer and film-maker, is unusually relieved because the van has been his companion for thousands of miles while exploring Britain’s “stopping places” – the sites where Travellers traditionally pulled over – and the journey has become the subject of his first book. But his relief may also be partly because the Transit is one emblem of modern male Traveller identity, and he is alive to accusations that he is somehow not a real Gypsy. It has been suggested that Le Bas, who went to boarding school and Oxford University, is merely a hipster with a van.

“A publisher asked: ‘Aren’t you just some really middle-class guy who happens to have a Traveller background?” says Le Bas. “He was implying that because I’m educated, I’m inauthentic. That is not a fantastic thing to say – that you can be educated out of your ethnicity. You can’t be educated out of being a Jamaican.”

Le Bas is wearily familiar with the casual racism towards the ethnic minority of Gypsies and Travellers. As the former editor of the Travellers’ Times, he devoted himself to defending the different Roma communities, particularly after news stories about Gypsy criminality or disputes when Travellers settled in villages. He found this “exhausting and demoralising”, and wanted to explore both his own roots and the history of Gypsies in Britain, so his road trip began.

Life on the road was a novel experience for Le Bas because, like many contemporary Travellers, he grew up “settled”, just outside Hove. “It was a typical Travellers’ yard in that there was an antagonistic relationship with the local authority. They had distributed flyers prior to my family moving up there saying: ‘Stop the Gypsies moving in.’” Le Bas spent his early childhood in houses, mobile homes and caravans (“trailers”) with his extended family, including his mother and father, who were both artists, and his nan – great-grandmother Julie Jones, who is 91. We meet at her house in Worthing, and she listens with interest as we talk in her front room.

When Le Bas was 11, he won a life-changing scholarship to Christ’s Hospital, a boarding school in West Sussex. “My grandfather was very against the idea of me going to this weird school where who knew what I’d be exposed to. When I was homesick, my dad just wanted to pull me out. My mum stuck to her guns. It was very hard and upsetting for her but she thought if I went to a local school I might end up in trouble.” What did his nan think? “I cried when he went but it was a good thing,” says Jones. Was his family worried Le Bas would abandon Gypsy traditions? “No,” she says. “I thought of him being safe there and doing good for himself.”

Le Bas didn’t feel safe enough at boarding school to reveal his background to any pupils except for two of his closest friends. “And also I wasn’t sure what it was. A lot of my family are true Romany Gypsies but I didn’t know where I fitted in to all that. I felt like I belonged to something but Travellers aren’t always so sure.” Although Le Bas speaks Romany, there were non-Gypsy relatives on both sides of his family tree. “A lot of Travellers draw a hard line between Gypsies and gorjies, non-Travellers, and I’ve got both in me.”

Le Bas went on to Oxford, where he studied Hebrew and Greek as part of his theology degree. How many Gypsy students did he know? “I’ve never met another Traveller who has been to Oxbridge,” he says. Even though Le Bas was an exception at university, he grew more confident about his identity and challenged his peers’ racism. His education made him an exception among Travellers, too: he didn’t marry until he was 26 (his wife is the actor Candis Nergaard), unlike his mum who was married at 19, and he acquired an unplaceable accent, not at all plummy but with a Kentish inflection that sounds older than his years.

It would be easy for The Stopping Places to be an angry rant, but Le Bas consciously “killed my old journalistic self” and tried to “eschew romanticism and angry activism” to write an elegiac contemporary travelogue. The book is also distinctive for being arguably the first literary exploration of Anglo-Romany culture by a Traveller. Serious studies of Gypsy culture are mostly the preserve of non-Gypsy academics (with a couple of exceptions in the United States). “It has been a monopoly of non-Gypsies for ever, to be honest,” says Le Bas. The book was also a more personal challenge: “I wanted to test the idea of being ‘a lesser son of greater sires’, a phrase from Tolkien,” he says. “It’s not easy growing up with stories of Nan’s father, who could weave a basket from reeds that could hold water. And master horses and heal people with herbalism. That’s how skilled he was. What can I do? I spent my childhood playing a Game Boy and riding a BMX.”

Le Bas set out in his Transit to visit the historic stopping places remembered by his nan and others: commons, churches, laybys and former fairgrounds. He was met with friendship and, occasionally, suspicion and hostility, including an uncomfortable encounter at Appleby horse fair. When Le Bas pitches up with Candis, a drunken man mistakes him for a new age traveller and then threatens him. As Le Bas writes: “We’ve driven thousands and thousands of miles and stopped in atchin tans [stopping places] from here to the Mediterranean, with hardly a tale of trouble to be told. Yet it’s here, at the English Gypsy’s Mecca, a place where Dad, Grandad and Nandad have trod before me for 50 years, that a Travelling fella strolls up to put the mockers on my week.”

But Le Bas soon becomes more comfortable in his own skin, and on the road. It changes his perception of time, he’s more aware of nature (“I’m not saying I’m some kind of horse whisperer but I’d be a little bit more attuned to what animals were doing”) and realises the starkness of the distinction between summer and winter. “Summer is really nice and winter is really horrible,” he says. Mud besmirches his van’s tiny, neat living space. “You’re hemmed in by rain for days. And if you’ve got washing to dry, it’s hard. Don’t ask me – Nan lived it.”

During his travels, he developed a conviction that we should recognise a Romany geography. One stopping place, Messenger’s Meadow, had a sign instructing people not to disturb the soil because of Roman remains. “If there are Roman remains in the soil and we shouldn’t disturb them, we might want to think about the Gypsy remains at old stopping places. Roadside burials used to be common when people couldn’t afford a formal interment.” Not only are historic stopping places unheralded, but they are often barred by bollards or bunds of earth to stop their use today. Le Bas only knows one common in southern England on to which Gypsies can still freely drive. Nearly a decade ago, there were a series of conflicts when Travellers bought land and “settled” in rural villages. “People always point out: ‘They are Travellers, they should travel,’” he says. “It’s not the travel that’s the problem, it’s the stopping.”

Is intolerance towards Travellers growing? “It has been pretty bad in the past,” he says. “At the minute, they are not hanging Gypsies for being Gypsies in England.” Le Bas writes about the case of Francis Buckley, a 16-year-old Gypsy boy from Islington who was arrested, tried and hanged, bizarrely, for impersonating a Gypsy – “counterfeiting himself to be an Egyptian”. Romany people, Le Bas argues, are “permanently trapped” in “the pincer of demonisation and romanticisation”. He dates the rise of anti-Gypsy prejudice to the birth of the nation state. “The nomad presented a problem to the nation state in terms of collecting taxes, because they had no fixed address. But there is also fear of a group that is perceived to be clannish and apart in some way. Many of these prejudices are very similar to antisemitic prejudices, and they are not really about nomadism. They are about a group that exists simultaneously within and outside European society. You can’t disregard pure racism as being a factor.”

Now the internet amplifies hate, says Le Bas, and the mainstream media doesn’t help. At the Travellers’ Times he tackled the fallout from the Channel 4 series, My Big Fat Gypsy Wedding. “It did an awful lot of damage, not just to Travellers, but to people who made the programme and the people who consumed it.” He’s still enraged by its “colonial tone” and the myths it propagated, such as that there is a tradition of “grabbing” young women. “That was some teenagers in a car park outside Staines bragging about their exploits and they inferred that this was a cultural practice with deep roots, which is absolute rubbish.”

For all his determination to challenge prejudice, Le Bas does not seek to “represent” his people. “In so many ways I’m the least representative person you can think of,” he says. Besides, Travellers do not want to be represented. “There is no Board of British Romanies, nor is there likely to be. But I felt I had a responsibility as a thinking citizen as much as a Traveller to challenge this deeply unsatisfactory media narrative, which lacks nuance and colour. It’s just so boring and repetitive and unfair.” That narrative focuses on criminal or antisocial behaviour. “Journalists can’t report on an absence of criminality, on an absence of rubbish, because there’s nothing to report on, and so those who behave badly monopolise the coverage,” says Le Bas.

In the future, Le Bas hopes he might articulate his culture through fiction but he also wants to defend the Romany language, which still has no official recognition in British law. “I worry that kids aren’t learning the language to the same level,” he says. “A view of the world can live within a language and be lost with it.”

Le Bas’s writing may be indebted to the rhythms of spoken Romany but “there are a few people who told me in no uncertain terms I shouldn’t be writing [the book]” he says. Within his community? “Within my family. I don’t speak to them now. I have to believe that writers have to write.” The opposition to his book comes from a Traveller suspicion that books about their people tend to be salacious exposés and “that general feeling that you shouldn’t tell the gorjies about our culture”. says Le Bas, “but not everyone shares that and hopefully they will realise it’s not a salacious book”. He is no longer anxious about the reaction to The Stopping Places. “As long as Nan thinks it’s all right I don’t care.”

And what does Nan think? “It’s good,” says Jones. “I tell him, if he wants to do it, get on with it. You can’t live your life for other people. You’ve got to do what you want to do, what you think is right.”

Le Bas smiles. “You see, Nan just sums it up better.”

The Stopping Places by Damian Le Bas is published on 7 June by Chatto & Windus, priced £14.99. Guardian Bookshop price, £10.99.

• This article was amended on 13 June 2018 because an earlier version referred to internment when interment was meant.