Caryl Phillips, 66, was born in Saint Kitts and raised in Leeds. The author of 12 novels, including 1993’s Booker-shortlisted Crossing the River, he lives in New York and for the past 20 years has taught creative writing at Yale University. He and I met on Portobello Road in Notting Hill, the location of his new novel, Another Man in the Street, in which a young West Indian finds himself collecting rent for a 1960s slumlord.

Tell us how this book began life.
A few years ago I was wandering around these streets, thinking it doesn’t look like the place I used to wander around as a student: no reggae shops, no guys on the corner smoking dope. It’s where Beckham lives! David Cameron’s got a house here. I began to think about how Notting Hill changed, and the nature of that change, and my own relationship to this gentrified, almost theme-park area of London that years ago meant something entirely different to me.

Which of the novel’s characters came first?
The landlord. In 2019 I wrote an article about the slumlord Peter Rachman, essentially because I’d never heard a West Indian say anything bad about him, even though his name’s become a byword for rapacious behaviour. The more I learned about him, the more I realised he’d got a raw deal in the press. It was anti-immigrant sentiment and antisemitism. I wanted to investigate his story as a Polish Holocaust survivor who ended up giving accommodation to immigrants at a time when nobody else would. Doing that piece, I began to think imaginatively about him, and the more I’m asking myself empathetic questions of interiority that go beyond what someone did or said, the more the pendulum tips toward fiction.

Is that what fires you up as a novelist – the thought that someone’s story might be more complicated than we can know?
I’m interested in the baggage people bring when they arrive in a place, the hidden baggage they can never set down. Not just in a West Indian context – I grew up in a city full of Irish and Jewish migration, and one of the biggest journeys I ever made was coming from the north to London [in 1980]. I used to think migrancy put you at odds with the main narrative, which for me, growing up, was rootedness – knowing your grandma, your grandad, your uncles, your aunts: I thought I was a freak for not having that massive extended family in Yorkshire. As I’ve got older, I’ve realised migrancy is the main narrative. When I walk out my front door I’m looking right at Ellis Island and the Statue of Liberty: it reminds me every day that people come from overseas with hidden narratives.

You recently wrote of how uneasy you were to be named on Granta’s once-a-decade list of best young British novelists back in 1993 …
I watched a number of my contemporaries fall victim to celebrity – I’m not going to name any! It’s seductive to be spoken about, to be profiled, to be part of the zeitgeist, but the worst place for a writer is centre stage with lights in your face: you can’t see shit. I was lucky that the first real writer I ever knew was James Baldwin. He was a bit of a celebrity hound – an incredibly generous man, but he loved the lights. I was able to see that it can be very damaging.

Was moving to the US a way to dodge that?
Originally, yeah. Every time a black geezer threw a bottle at a policeman in the 80s and into the 90s, some bugger from the BBC or the Guardian would ring me up for a quote. “What do you mean: ‘What do I think of that?’ I don’t think anything of it!” Anonymity’s good. You need to do your work without feeling the pressure to be a spokesperson. It wasn’t a huge pressure – I’m not suggesting everybody was interested – but it was nice to just work. I was never going to lose touch with my concern for British society, because it was Britain that made me a writer. All the anxiety was inculcated into my soul on the terraces of Elland Road! America didn’t do that.

Do you still get to watch Leeds United play?
Every game. When I first went to America to teach for a term [in 1990], I had to listen to the World Service to hear the second half. Now every match is live on TV. The game at the weekend was a 7.45pm kick-off, 2.45pm in New York. For a key game I’ll go to a bar in midtown Manhattan where Leeds fans go – it has a big “New York Whites” banner – but otherwise I’m not really a crowd person, so I’ll watch on the sofa with my two sons in their Gnonto shirts.

How often do you return to Leeds?
Very frequently. I’m writing something at the moment about my old history teacher, a guy who got out of Berlin in 1939. He’s probably on my mind because of this novel. So I’m up there quite a lot, trying to find out as much as I can about the fact that he founded a synagogue in the middle of what is now the black community. All this stuff I never knew when I was a 12-year-old – I just thought he was a mean bastard who used to give me lines and hit me across the head. Now I realise he did that because he cared: he spotted another immigrant.

Name a writer you enjoy reading.
Barney Ronay. If I’m picking up a book, it’s work.

What novel did you last assign your students?
One of the best I’ve ever read: Reading Turgenev by William Trevor, about a young Irish Protestant girl in the 1950s in a village in which the Protestant community is dying out, and she therefore makes a very ill-advised marriage and suffers for 30 years because of it. I love the structure. You ask yourself, how am I going to write anything that good? I actually think it’s a really important thing to feel – that the bar is up there, you know: how am I going to do it?

• Another Man in the Street by Caryl Phillips is published by Bloomsbury (£16.99). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply