This is the year of David Nicholls. Which has come as a bit of a shock, it turns out, to David Nicholls. He had not planned to release his sixth (and, he thinks, his best) novel You Are Here at the same time that the adaptation of his literary phenomenon One Day hit Netflix, or as a musical of his 2003 novel Starter for Ten opened in Bristol. “In one way, it’s very exciting. But in another way, it’s a bit…” he says, searching for a word that is kind, “overwhelming.”

The year of David Nicholls means this is a year of the bittersweet. Of love as a matter of life and death and the path from friendship to romance, and all the tiny, exquisite, effortless banalities of modern life that have seen his work, which has encompassed film, telly (including Patrick Melrose, the Emmy-nominated adaptation of Edward St Aubyn’s novels) and particularly his novels, reframe what the truest story of true love could be. Rather than romcom, one reviewer described his work as “rom-trag”.

When Nicholls’s agent, Jonny Geller, first read one of his manuscripts 20 years ago, he “felt instantly that he had an uncanny ability to write stories that seem to reflect my life – as if he had been in the room”. Geller soon realised he was not alone in that feeling. “It’s not just the comedy but the observations about disappointment, frustration, yearning and trying to be a better version of oneself that makes his stories universal.” Nicholls’s books have sold more than 9m copies worldwide, in 40 different languages. One Day alone has sold more than 6m worldwide and, 15 years old now, is currently back in the Top 10 charts after the success of the Netflix adaptation, which in its first month was watched by over 15 million people, one of whom was Kim Kardashian. She recommended it to her 364m followers on Instagram, saying: “If you want a good cry.”

We are meeting for lunch near his house in London, where he’s lived for 20 years with his partner, Hannah, and their two teenage children. Recently, though, he has been making a regular journey up to the mountains or the coast, where he will walk, by himself in weather-appropriate clothing for, he says, brows lowered, “time to think”. In his new novel, two lonely people fall in love on a 200-mile walk across the north of England – for Nicholls, however, it is crucial he walks alone. He says it’s time to think, yes, “but, you know, you can think on the beach, can’t you? You can think somewhere warm. I don’t know what it is about the arduousness of it that’s appealing.” He allows himself a moment of bewilderment, gently spearing some cheese on his fork. “I’ve always hated sport. Anyone throwing a ball at me, I feel, is hostile. That thing of wanting to beat someone. I wonder if walking is just the closest I can get to a sport in my life? I don’t quite understand it, except that a certain kind of melancholy and loneliness is part of it, and I would rather be by myself.” Does he come up with ideas while walking? “I always think I should. I have a little waterproof notebook. It just looks like a regular notebook, but the pages don’t curl when they get wet. I’ve carried this notebook with me for 10 years and it’s got nothing in it at all. Ha!”

His first real attempts at writing were letters. Now 57, Nicholls was born in Hampshire and, after university (he was the first person he’d known who’d been), won a scholarship to study drama in New York. When he returned to England he worked as a bartender while auditioning for acting roles. “There’s a line in One Day where Dexter says that he wishes he could give Em the gift of confidence, which is something a friend of mine said to me then.” This was London in the early 90s and, “I think I was having quite a bad time. I was trying to be an actor, but often it was quite humiliating. It was lonely, really, looking back. And this is the time where everyone was meant to be, you know, out at some rave on the M25 and I really wasn’t.” What he was doing was crafting long, poetic, funny letters to his friends and “making the worst cappuccino on the King’s Road”. He was worried, “All the time. About the future, worried about what I was going to do, worried about money.” He didn’t let his parents see his flat – a bed and an electric hob in a very small room – he was too ashamed. “But they were worried, too. And confused, because I’d gone to university – it wasn’t meant to be like this.”

He got sporadic acting work, but knew he didn’t have what it took to be great. “I was too keen and too self-conscious. I was always putting on a voice, always doing too much. Just too kind of puppyish and keen to please. Which is why I got work. But if I sat and watched the really great actors…” He starred in The Seagull at the National Theatre alongside Judi Dench, Bill Nighy and Helen McCrory. “They just seemed to be doing something I couldn’t do. They were very still, they were very calm. They were very physically free and yet in control.” He was not. It occurred to him, around 1996, that while he didn’t really know what to do on stage, he did know what to do to a script to improve it.

“I wasn’t always right, but I felt on safer ground.” Aged 27 he was offered two jobs – one was to understudy for a part in Twelfth Night and the other was as a script reader for BBC radio. He chose the latter and gave up acting for good. “I realise now I learned so much from acting and from listening to plays being performed night after night after night. I used to feel quite bitter and angry, that those were wasted years. But looking back now, I learned a massive amount just from being in the room. I’m more philosophical about it today.” After some success as a scriptwriter on Cold Feet, he had two shows cancelled and, spurred on by rejection, at 37 wrote his first novel.

A kind of fame arrived in 2009 when One Day, his third novel, which meets Emma and Dexter on the same day over 20 years, was suddenly responsible for millions of people weeping on hundreds of trains. Does Nicholls feel defined by it? “It would be an absurd thing to complain about,” he says, carefully. “I’ve been very lucky with it, but when people say, ‘I enjoyed your book’, I know they’re not talking about Us or Sweet Sorrow. And at the same time, I do think everything I’ve written since One Day is better! But sometimes there’s an extra ingredient in a story that you have no control over.” A whole new generation has now been gorgeously, horribly moved by the story all over again, thanks to the Netflix adaptation written with Nicole Taylor.

“I started writing the book 17 years ago, when I was in my 40s, so it’s a bit like looking at an old photo of yourself – you can still feel fond of that person, and yet, it’s very much someone else.” When the book was reissued was he tempted to go back and do an edit? “Yes! With every book! Maybe I will one day. But it would be like airbrushing old photographs, wouldn’t it?”

He is uncomfortable even with the gentle celebrity that comes with being an author whose name is a kind of shorthand for complicated love, and rather than discuss his family or politics, online he uses his social media mainly to promote other authors. “I wonder,” he says carefully, “if it’s better not to know too much about what a novelist feels?” Those who read his novels should be able to divine his politics, he says, but if he were to speak directly to something, he would focus on education. “I get very angry about that. Libraries closing, the way the arts are not accessible. That kind of thing makes me really furious.” He doesn’t mention, though I discover it later, that he established a bursary to support theatre students at the University of Bristol, regardless of their financial circumstances. “I have personal experience of what an education can give you and I get angry when it comes under attack. It changed my life – being paid to go to university and having access to public libraries and local theatres. And I find the way that that’s been taken away from people like me at that age to be enraging. Going to university specifically to study something that isn’t vocational was absolutely life-changing.” Where is he leaning, politically, today? “Well, none of the political parties are suggesting a return to student grants or means-tested bursaries. And if they did, then that would be a factor in my support.” But, he smiles, a little tightly maybe, “I do feel excited at the prospect of a change of government”.

The first review for Sweet Sorrow, he remembers, “was a real massacre. Really spiteful and mean. But what it said,” what stuck, “was that all of the books were about nostalgia.” Sweet Sorrow was set in 1997. One Day begins in 1988. “Starter for Ten is a 1985 nostalgia-fest. Even Us, which was a contemporary novel, had flashbacks in it. And I was aware that a lot of what I was writing was about the gap between then and now, which I suppose is another name for nostalgia. A kind of… sense of regret,” he says, a little wistfully, “things going wrong in the past having an impact in the present, or a longing for the past.”

The review weighed on him. So when he sat down to write this book, “I thought, ‘You must set it here and now’.” You Are Here sees two people entering middle age, both disillusioned by love and both balancing the joys of solitude with a fear of loneliness. “It comes out of the feeling of leaving lockdown. That self-consciousness and awkwardness and tenseness that we have, and the way that we all questioned things, like the value of friendship. What had we lost?” Though this book is very funny, there is also that aching thread of melancholy, the stuff of Nicholls’s long cold walks. “Even in the funnier books that I’ve written, there are quite bleak moments. There’s a particular kind of comedy that’s just joyous and lighthearted. And I’m not sure I can do that.”

What does melancholy mean, to him? “There’s a kind of ruminative aspect to melancholy, a kind of self-reflexive, thoughtful quality to it. It’s maybe gentler than sadness, and there’s a kind of perverse joy to it, too. A weird, strange kind of pleasure in it, isn’t there?” He’s talking about the secret to his work, the sour that spikes the sweet. Is that why he seeks it out, slipping into his specialist socks, stepping out with his waterproof notebook? “This is really unhealthy for me,” he says, with implied apology, “but I don’t have an answer. When you’re an adolescent that whole business of being mysterious and solitary is very self-conscious, and I don’t think I’m particularly like that. I’m quite sociable, but I’ve always definitely slipped into quite long periods of sadness and anxiety.” He pauses. “This is a book about loneliness and I didn’t have to think very hard about what that’s like.”

He says this is a book about loneliness, but it is also about the pleasure one can find in being alone, an unusual theme for a love story. “I didn’t want it to be too downbeat about solitude or loneliness. Often in a romantic comedy, the temptation is to make the state of being by oneself just awful, something that you have to escape, just the worst possible set of circumstances.” But both Marnie and Michael find comfort and pleasure in their own company before they fall in love, which somehow feels faintly radical. “I didn’t want it to be one of those books where being by yourself is pure hell. Sometimes it is for them, but for a lot of the time, they’ve managed to have quite rich, satisfying lives that just don’t get romantically involved with other people.” Which, when reading, feels exciting, especially from someone who has come to define romance. “I’ve never been able to write a happy ending,” he says, but admits this is the closest he’s come.

When Sweet Sorrow came out, a journalist asked him what he’d like to do next, “and I said, ‘I’d love to do a very classic, contemporary love story.’ And they said, ‘Aren’t you a bit old for that?’” He chuckles and swallows a forkful of pasta before adding, “looking back, I think maybe they had a point!” He loved seeing the musical adaptation of Starter for Ten, but found it “very strange. There are things I’d written where I thought, where did that come from?” Like what? “There’s quite a moony-eyed love story, a lot of yearning from a distance.” It was interesting, then, for him to watch the musical one week, then start press for You Are Here the next. “Which is about love at a different stage of life. It’s much more thoughtful and gentle, a bit more grounded. A love story that is about experience.” He realised he has been writing about each stage of love as he passed them, from Sweet Sorrow, which is about being 16, and reaching, and dreaming, to Us, which is about facing divorce at 58. “It’s interesting to me that love is a different thing at every stage of your life. And the notion of love [in Starter for Ten] is to me now almost unrecognisable as love. It’s something else, to do with books and records and lust. It’s a strange kind of cocktail of what it’s like to be 18. Whereas Douglas’s sense of love in Us is much more mellow and poignant, and urgent in a different way.” Nicholls has come to understand that he’s interested “in the way in which love changes, depending on age and circumstance.” And looking at his body of work as a whole, “ I think the books are all about this great gulf between how love is portrayed for each of those characters, and how love feels for each of those characters.” But something new is coming. He is planning to concentrate more on film-making before settling down for his next novel. And, “next time, I don’t think I’ll write another love story”. I wonder, briefly, how we will cope. We’ll be fine, I say brightly. We’ll be fine!

He’s discovered how love changes with age – I wonder how he thinks love itself has changed since he started writing. “What the internet means for romantic relationships is completely inconceivable to me,” he says, wide-eyed as we share dessert. “But I do think there is more to it. When I think back to being 16 or 17, relationships were like a school disco with the boys on one side and the girls on the other. No one really said what they felt. I see less of that in my children’s lives. There’s a frankness now that is more healthy.” People of all ages have become, he says, much more clear about what they want, and how they feel. “My son always signs off his phone calls with love. It’s a very particular meaning of the word,” he says, smiling as only someone who has rewritten this emotion in ways that have helped a generation define it for themselves can, “but – I like to hear it.”

You Are Here by David Nicholls is published by Hodder & Stoughton on 23 April at £20 (Guardian Bookshop £17.60)