With the neat timing of one his books, David Nicholls’s new novel Sweet Sorrow is published exactly 10 years after One Day, the blockbuster romantic comedy that made his name and the rest of us weep. For a long time, the orange book jacket was the brightest thing on the morning commute and an obligatory stripe on the bookshelves of anyone under 40. Set on St Swithin’s day over the course of 20 years, it was a genuine word-of-mouth success; a film starring Anne Hathaway and Jim Sturgess as will-they-won’t-they friends Emma and Dexter came out in 2011. Since then Nicholls has written two big-screen adaptations (Great Expectations and Far from the Madding Crowd), and his fourth novel, Us, which was longlisted for the Booker prize in 2014. In May he was awarded a Bafta for his TV screenplay of Patrick Melrose, which starred Benedict Cumberbatch. He’s currently getting up at six in the morning to deliver scripts for the TV dramatisation of Us, which begins filming later this year. Not that he’s complaining – “I am so lucky,” he marvels several times during our conversation.

Despite all the accolades, not to mention dizzying sales figures, he is as self-deprecating as in his earliest interviews. Now 52, he seems as genuine and gently witty as you’d expect from his fiction. In his Bafta acceptance speech, he thanked the producers for “miscasting” him as a writer for Patrick Melrose. Although Edward St Aubyn’s blistering autobiographical series, a bleak history of abuse as a child and later drug addiction, had been his “dream project as a scriptwriter” for many years – he first encountered it when working in a bookshop in Notting Hill back in the 90s – he wasn’t, he says now, “an obvious choice” for the job, a bit like signing up Richard Curtis to make The Handmaid’s Tale.

Part of the appeal of Melrose was that it is “a sort of anti-romcom. For me as a writer there’s very little love in it.” But while it was “exciting and exhilarating”, it was also “completely overwhelming”, and after more than five years locked in St Aubyn’s feverish aristocratic world he was ready to return to more familiar, sunnier territory.

A story of first love, family breakdown and lost innocence, Sweet Sorrow is typical Nicholls: a little bit funny, a little bit sad with lashings of nostalgia. Set just before the turn of the millennium, it centres on a summer school production of Romeo and Juliet that brings together bored, lonely 16-year-old Charlie Lewis (a reluctant thespian with a troubled home life) and the lovely Fran Fisher. While One Day was his farewell to single life, “a hymn to friendship and growing up”, Us, about an ill-fated family trip across Europe, was his “mid-life crisis book”: his father died halfway through writing it, changing its focus from marriage to fatherhood. Nicholls’s children (Max, now 13, and Romy, 11) were very young at the time, and he was preoccupied with the responsibilities of being a parent, “how to do it well, the mistakes you make”. Now Sweet Sorrow is an affecting combination of both: the youthful romance of One Day and the darker, adult failures of Us.

“I must stop using the word ‘bittersweet’,” Nicholls says. But bittersweet is what he does best, along with a gift for making the reader feel young. Like all the most memorable coming-of-age novels, One Day pulled off the feat of seeming to speak for a generation while being written just for you. And, if Us failed to deliver the same body blow – you can’t pull off that sort of trick twice – it was perhaps because it was hard to love its pernickety, middle-aged narrator. “Us was quite a sad angry book,” he says. With Sweet Sorrow, he was happy to write from a young perspective again. He conceived of it as a 50,000 word novella, “a little sweet summer story,” in the tradition of Alain-Fournier’s Le Grand Meaulnes, but he enjoyed writing it so much that it “sort of sprawled a bit”.

“That time of life is characterised as wild and carefree, and actually that is very rarely the experience,” Nicholls reflects. “I wanted to write a teenage love story that had the booze and the sex and the kind of gaucheness and the passion for music and culture, but also had a sadness to it, confusion and a kind of romantic tender quality.” (“Tender” is another favourite Nicholls word.) His own summer waiting for exam results was spent working in a factory making coffee percolators, the highlight a three-day cycling holiday on the Isle of Wight with his friends Baz and Neil: “There was no great romance, no falling in love. It was really dogged and dull.”

Nicholls turned 16 in 1983. His debut novel, Starter for 10, about a working-class boy obsessed with getting on to University Challenge, was loaded with 80s nostalgia. It was picked up by the Richard and Judy Book Club in 2004 and Nicholls later adapted it into a film (with a small part for Cumberbatch as another, very different, Patrick). While Starter for 10 was strongly autobiographical, with Sweet Sorrow he wanted to create some distance from his own past, to avoid writing “the novel version of your own photo album”. It looks back to the summer of 1997, bookended by the election of New Labour and the death of Princess Diana; but unlike One Day, which, as he says, is “like flicking through old newspapers”, you have to be paying close attention to catch references to these landmark events. “It felt like one of those big years that come round two or three times a decade, a year in which lots of things are changing, in which the culture seems to have a particular resonance.” This is the era of Britpop: Blur and Oasis are there, the Spice Girls at the school disco (“a kind of musical water-cannon” to get the boys off the dance floor). If the novel has a soundtrack, it belongs to Pulp, in particular “David’s Last Summer”, “a brilliant song, a poem spoken to music” about the last weekend of the summer holidays, with its “kind of cider in the park mood” that he hoped to distill on to the page.

And of course 1997 was still pretty much pre-internet – “maybe the last year you could make a mixtape” – and all that means for writing a love story: if only Romeo had had an iPhone rather than a dagger in his back pocket on the way to Padua. Surprisingly perhaps, Sweet Sorrow recalls last year’s hit novel Normal People, another teenage romance, but with more sex and texting. Sally Rooney has been called “the Salinger of the Snapchat generation”, but Nicholls is happy, for the moment, to leave digital speak and social media to the natives. “‘Tweet’ feels like quite a clumsy, ugly word that draws attention to itself,” he says, although he knows he is going to have to “grapple with all that” at some point. One thing that hasn’t changed is the awkwardness of growing up a sensitive, clever boy in a small suburban town, the strict codes, aggression and rivalry of male friendships, so touchingly portrayed in both novels. “To not be a dick”, as Charlie puts it, “that was the great rite of passage.”

“It feels like my most personal book, without being autobiographical,” Nicholls says. “It’s the one I feel the most emotionally attached to.” He grew up in Eastleigh, Hampshire, which is not unlike the unnamed commuter town in the novel. His father was a maintenance engineer at the Mr Kipling cake factory and his mother worked for the council. Despite the acuteness with which Nicholls writes about divorce, his parents were happily married. “I think so,” he replies cautiously. “It was stable.” Father-son relationships (all fairly disastrous) have been a preoccupation in his work – from his 2007 adaptation of Blake Morrison’s memoir And When Did You Last See Your Father? to Great Expectations and Patrick Melrose. He is quick to stress that his own father was nothing like the troubled dads in either Us or Sweet Sorrow, but “the difficulties of the bond, fraught with communication pitfalls, is something that I’ve come back to,” he reflects. “It’s something I think about a lot with my own son, and my memories of my father.”

A middle child, he has described his upbringing as “a sort of Narnia and Coronation Street”, in which he would be reading while the rest of his family watched TV. Unlike Charlie, he was “a real swot. I poured everything into schoolwork.” He was also “a comedy nerd”, given to quoting Monty Python and The Hitchhiker’s Guide to the Galaxy: “I was a class clown who couldn’t make anyone laugh. I was just trying a bit too hard.”

In his first year of studying English and drama at Bristol University, he spent a “very memorable summer” putting on a production of A Midsummer Night’s Dream in the grounds of a country house. (More prosaically, he also draws on his experiences of working in a petrol station at this time: Charlie gets into a painful scrape involving free wine glasses.) Like Charlie, Nicholls felt, “and I suppose still do a bit”, a sense of not quite belonging in that privileged world, “that the culture was both interesting and quite excluding”, a feeling that became overwhelming when he went to New York to study acting: “It was very Broadway oriented and I can’t sing or dance. I thought it would be this exciting baptism of fire and it was actually just quite lonely.” He took consolation in writing “very long, very ‘written’, sweated over” letters to friends back home – his first attempts at comic writing.

He suffered a “cliched case of twentysomething anxiety”, of not really knowing what he wanted to do with his life. Initially he had ambitions to become a standup comedian (“a terrible idea. I didn’t have the swagger”) before falling into acting, which in reality meant years of small parts, bartending and some starry roles as the understudy. As he wrote in these pages: “I’d committed myself to a profession for which I lacked not just talent and charisma, but the most basic of skills. Moving, standing still – things like that.”

While he loved the intensity and sociability of the theatre, “the whole business of putting on a play”, the real appeal was that “they are written, they were characters and stories and jokes and they had an effect on people”. And while his acting training “was a complete waste of time” in terms of becoming an actor, it was a good apprenticeship for writing fiction, teaching him “basic tricks like, ‘what did my character have for breakfast, and what’s my favourite song?’” After some script-editing, turning down a minor role in an RSC world tour for radio work, he was asked to write for the hit TV comedy drama Cold Feet. “I kind of owe it all to Cold Feet,” he says now. It was just his second produced script, after co-adapting Sam Shepard’s Simpatico, which also appeared in 2000, and he only wrote four episodes. For the record, he didn’t kill off Rachel – that happened a couple of series later.

Though scriptwriting “can be quite architectural and technical”, it provided Nicholls with invaluable training in terms of plotting and pacing, sometimes lacking in more obviously “literary” fiction. He quotes a piece of writing advice: “Decide the effect you want to have on an audience and then do everything you can to achieve it.” So if you set out to break the nation’s heart, the challenge is how to do it “without being hokey or sentimental or predictable or mawkish? How can you do it in a way that isn’t cliched or familiar?”

“It doesn’t matter what else I write now or for the next 20 years, I know I will always be the author of One Day,” he says, without resentment. He feels he would be “too self-conscious” to write it now. Does it ever feel like a millstone? “It did a little while ago,” he admits. But now he has “nothing bad” to say about it, except an urge to “tweak it a bit – to take out a couple of pages, there’s lots of gaucheness in it”. After a couple of years of false starts, he scrapped 30,000 words of his next manuscript – “a kind of perverse reaction … it was quite misanthropic and rancorous” – before beginning on the novel that would become Us. And though he knew it would never be as big a hit as One Day, making the Booker longlist was “a complete career highlight”. He was also prepared for the inevitable raised eyebrows at his inclusion. Does such snobbery bother him? “If you sit down at your desk and think ‘I’m going to become a little bit more literary today’, then you end up writing pretentiously,” he says carefully. “At the same time, if you sit down and think, ‘Well I’m not going to deal with that because it’s a bit dark or difficult or it’s not necessarily crowd-pleasing’, then that is just as limiting. So I genuinely don’t think about it.” But, he adds, “that doesn’t mean literary approval isn’t important”.

If his books are getting sadder and darker, and “better”, as he says with uncharacteristic assurance, it is perhaps because of his immersion in the gloomier worlds of Hardy and St Aubyn. Adapting other novelists’ work has given him the courage, “a sort of leg up”, to take more risks and approach more complex issues (mental illness in this latest novel, for example: “You look back and think, ‘We didn’t call it that, but it was that’,” he says). But he doesn’t “feel the need to write the great serious novel” and, happily, has no intention of abandoning the genre for which he is best loved. “Could I ever write a book that had no romantic comedy elements in it at all, no love story? I don’t know if I could, or if I’d want to. It is such an interesting, life-changing event in most people’s lives. There’s no reason why it can’t be given weighty treatment.”

Although “very flattered” by comparisons with comic – male – British writers such as Jonathan Coe and Nick Hornby, he feels he has been more influenced by contemporary American writers such as Lorrie Moore and Elizabeth Strout, as well as the 20th century heavyweights, JD Salinger (“brilliantly funny”), James Salter and Philip Roth (Goodbye Columbus was a particular inspiration for Sweet Sorrow). He doesn’t “feel part of a literary set” or spend his time hanging out with Cumberbatch. He finds going on set “quite frightening and stressful. You can’t help them move the lights, you just feel like a spare part.” And, despite arguably being at the top of his game, he is forever expecting it all “to come to an end at some point”. He is “quite anxious”, he admits. “And I don’t think that will ever change. I used to worry about the fact that I worried.”

As Dexter tells Emma, in a much quoted line, if he could give her one gift for the rest of her life it would be “Confidence … Either that or a scented candle”. If you had told Nicholls’s teenage, Hitchhiker’s Guide-reading self what his own future had in store he says he would not have believed it. “I had no expectation of it and would no more think that it would happen than that I would land on the moon.”

Today he works in an office round the corner from where he lives in Highbury in north London, with his partner of more than 20 years, Hannah Weaver, a script editor, clocking off at five to go home and cook tea. “There’s a kind of romance to the idea of sitting there with a whisky and writing until three in the morning, but I’ve got kids.” After the filming of Us, he is looking forward to a break and reading a lot. And then, hopefully, novel number six will start to take shape.

“I should perhaps try and write a book that is entirely serious, or entirely comic,” he muses. “But it seems to be, for me anyway, the natural way to tell the stories, the natural way to approach life and experience. That it is a mixture of comedy and pain. That sounds melodramatic or pompous, but it feels like the only voice I have.”

• Sweet Sorrow by David Nicholls is published by Hodder & Stoughton (RRP £20). Nicholls will discuss Sweet Sorrow at a Guardian Live event in London on Monday 8 July. To book tickets visit gu.com/live-events.