Richard Osman says that people will always tell you who they are, if you listen. So, I listen, and he tells me – somewhat urgently – that he is not a nerd. Nerd-presenting, perhaps. Most explicitly as the creator of TV quizshows and host of BBC One’s Pointless. Absolutely not a nerd. The evidence he supplies, twitching his shoulders and tugging his jacket as if to do it up, is that he does not like superheroes. He does not like sci-fi (despite, he jokes, marrying the Doctor Who actor Ingrid Oliver), does not have “nerd hobbies”, such as chess. Correction: he did like chess, he just wasn’t “very” good. So, how would he describe himself? “Testosterone-y, a huge sports lover. I’m fairly alpha at times.” Trouble is, this rippling machismo “is all hidden under a fairly gentle exterior”.

I take in the gentle exterior. He’s dressed in shades of blue, bloodhound features behind trademark specs. Minutes earlier, he told me, “I love statistics.” He explained his formula for “decoding” the world. He said, “I’m an alpha introvert.” Top of the sports he enjoys is snooker. He chose the theme tune from the BBC’s snooker coverage as one of his Desert Island Discs, has encyclopedic knowledge of the game and goes to the world championships. Perhaps I’ve misunderstood “nerd”. Perhaps Richard Osman is a man of contradictions. Here is another: behind him, through the window of this office in Westminster, is the MI6 building across the river. It sits as if on his left shoulder. Appropriate because Osman was given “the tap” while at Cambridge; took a series of “fun” spy tests which – ultimately – he failed. “They just said, ‘No, it’s fine.’” He did war-gaming scenarios, chatted to people “who got older and posher throughout the day. Honestly,” his voice develops an insistent edge, “I would have been terrible. I’m too tall [6ft 7in], not bright enough, and if I have a secret, I tell everybody. You could not find a worse spy.” Also: “I cannot tell a lie.”

And yet and yet, he writes fiction. Fiction so popular that the Thursday Murder Club series – the fourth, The Last Devil to Die is out next week – is publishing’s new Harry Potter, selling 5m copies to date and grossing more than £25m. He is the biggest new fiction author of the decade, the series has been optioned by Steven Spielberg, and he has been signed to write a separate (undisclosed) crime series for Netflix. Or as Osman puts it in his self-deprecating way: “I’m doing TV things. And film stuff.” Hasn’t Penguin, in whose offices we sit, just signed a new four-book contract with him reportedly worth more than £10m? “I was given … Oh! I’m not sure what to say the advance was,” he says, seeming bashful. “I’ve been given an amount of money which makes sense for Penguin and makes sense for me.”

Only two books of the deal will be Thursday Murder Club titles. He has begun another series with new characters. This should not upset fans, he says. After all, Agatha Christie had Miss Marple and Hercule Poirot. “That’s not me comparing myself to Agatha Christie, by the way, but you do have to [diversify].” For those few not au fait with Osman’s pensioner procedurals – perhaps you have not passed any bookshop windows in three years – The Thursday Murder Club is set in a retirement village. Osman’s sleuths, his “A-Team”, are Elizabeth, a former spy, Joyce, a former nurse, Ibrahim, a psychiatrist, and Ron, who fondly recalls his years as a rowdy trades union official. “They’re all quarters of my own brain. I’m most Ibrahim” – Ibrahim has a maths-based, practical outlook on life, enjoys data, laminating printouts, time-specific goals and is bad at noticing what people look like – “then a bit of Joyce, then a bit of Ron. I’m least Elizabeth. She’s who I’d love to be.” He says as an aside: “I’m still available, by the way, if MI6 read this. I could be useful, because no one is going to suspect me now.”

Rejected by the intelligence services, he graduated in 1992 and went into TV, working first at Planet 24, boisterous makers of The Word and The Big Breakfast. Colleagues remember him as “very quiet”. At Hat Trick Productions he wrote for Whose Line Is It Anyway? and Have I Got News For You. Later, he became the master of the TV format, working on, among other things, 8 Out of 10 Cats and Deal or No Deal. While creative director of Endemol, he sold Pointless to the BBC and ended up presenting it with his university chum Alexander Armstrong. There were failures too, among them 24 Hour Quiz and Boyz Unlimited, a music industry satire, starring a young James Corden. “Oh my God, I know what failure tastes like,” he says, shaking his head. “I also know it’s what you do next that matters. Always.”

By his late 40s, he’d made enough dosh and, anyway, was no longer “driven by sitting at the table and banging my fist on the desk”. So, he started The Thursday Murder Club – quietly, secretly – and showed not a soul (although halfway through he confessed to his brother, Mat, bassist in the band Suede. In turn, Mat confessed he too was halfway through a novel, which was a “weird” coincidence). In the years since, terrestrial TV has “fallen off a cliff. The world that I was in, it’s not there. The money’s not there. I have no home to go back to.”

Some might see the blossoming of this late second career as a thing of romance, an example that an entirely new life after 50 is an achievable thing. Osman is more prosaic. He draws a direct comparison between the mechanics of TV entertainment and the procedural format. “You know at the start that you’re heading somewhere, you’re at A and you are going to get to Z. You just have no idea how. I find it more creative to be given a framework, to be given constraints. As a TV host, I’m saying, ‘You don’t need to like me, here’s a show for you. Here’s a format. I’m going to take you in a direction. You answer some questions.’ And in a crime book it’s the same. You go, ‘Look I’m setting you a puzzle.’ Because otherwise, what is it [the novel]? Just me talking, which doesn’t interest me.” Osman says he is not the sort of person to write about “love and loss”: “I don’t feel like I am somebody who can sit down and describe what the sky looks like … the beauty of the summer flowers. Whereas I can write a story and move the action on. I’m very comfortable imagining worlds, imagining people, imagining what they might do.”

He puts the extraordinary success of the books down to an innate understanding of what makes Britain tick – cross-reffed with data. “The two school subjects I use every day are O-level statistics and my O-level sociology. Sociology tells you the world wasn’t made this way; it has been constructed by us through a series of choices. Statistics tells you the truth about things. With those two tools, you can pretty much decode anything, I think. Why people do things. What drives people.”

Decode? “I find it obvious why people behave in the way they do. But you can check you’re right by looking at the statistics.” Second source it, so to speak? “Exactly, just in case. But we don’t use statistics brilliantly in this country. My whole career is: this is what I think people will like and why.” He was always amazed in television to be surrounded by people for whom that wasn’t their natural instinct.

On the other hand, he believes there is often a gross distortion in the things we’re told everyone likes. The TV series Succession, for instance. It’s only watched by a teensy sliver of the population, but for all the amplification it gets in the news we’d assume it was a nationwide preoccupation. Ditto GB News. “Statistically, they’re insignificant when it comes to what’s happening in this country. Yet you would think from social media that those are the two groups of people fighting each other. There’s no one in those groups. It’s just that everyone in them are the people we hear from. So, I drive my bus straight through the middle and park it far away from both sides.”

In the real world, people are rarely as obsessed with politics as we are told, he says. He compares it with football: some people are fanatical, go every week, know all the players’ names, etc. Most tune in every couple of years when there is a big tournament.

Coopers Chase, the fictional retirement village where his characters reside, is based on the community in Sussex where he bought a house for his mother, Brenda. He looked around and immediately realised the potential. Here was the generation who, culturally, “are overlooked by everybody. That generation did much more interesting and unusual things, overcame much bigger hurdles and obstacles. It’s a generation full of wisdom, full of brio, looking for new adventures and new mischief. There are very few consequences to anything they do or say. That’s freeing for a writer, to have characters who are going, ‘No one’s going to arrest me; I might as well do this.’” They are overlooked because we worship at the temple of youth, he says. “I mean, God knows what our generation will be like when we get to that age; insufferable.”

Some of the best fictional sleuths are older and wiser – from Miss Marple to Columbo to Jessica Fletcher in Murder, She Wrote – but he cannot abide the term “cosy fiction”, words he introduces to the conversation and has a severe allergic reaction to without any input from me. “If you really read my books, there’s some quite bad stuff happening, some very non-cute references. It’s definitely not cosy. Today you can write a book about a detective who runs a sweet shop in a seaside town and someone will publish it. But that’s OK,” he says moving from the heat of his own irritation. “I get it.”

What also nettles are the copycat swirly font covers that have followed in the slipstream of the Murder Club’s success. “Richard Bravery created the cover – so great, so iconic. Now everyone does the same. We’re working on the next series. The two of us sitting there going, ‘We’ll show ’em. We’ll give them a different cover, a cover that makes them go: Ooooh, that’s what we need to copy now.’”

The new project is more Sherlock Holmes-y in that it revolves around an agency and is “more of a traditional detective thing. With the Thursday Murder Club, the crime has to happen to them, whereas as soon as you’ve got a detective agency, someone can knock on the door and give you your plot.” The characters are a father and daughter-in-law, who work on different sides of the world. “So, it’s slightly more internationalist. Hopefully the same wit and Britishness about it, but they can jump out of helicopters and kick down police station doors, which Elizabeth is not able to do.”

Osman always loved crime. He grew up reading Christie and adored Patricia Highsmith, creator of Tom Ripley. He also likes the peculiar Britishness of the worlds created by Barbara Pym and Muriel Spark. It didn’t feel like a jump to write books, having written for TV. Although he has been accepted with open arms by the crime writing community, there is still a trace of the testiness he felt over an early suggestion that he is one of a slew of celebrities turned authors. “There’s certain books that come out and people are open about having a ghost. I get that people know what they’re getting and understand it’s a brand. But there’s also a group of people – Bob Mortimer is one – where we’re just writers. I’ve written my whole career, my whole life. Graham Norton has always written, Dawn French has. It is not a surprise that these people go on and write books. You’re allowed to. Also: no one is a writer. Everyone is something, then becomes a writer. You get to a certain age and think, ‘Well, I want to write a novel. I’ve got stuff in my head that I want to say.’ No one ever buys a second novel if the first one isn’t good.”

Included in “stuff I want to say” are urgent issues. To Osman, it’s key that the books are funny – in the same droll way that he is – with issues smuggled in. “To know that you’ve written something that’s going to entertain people and while doing it given a family a way into a debate, a discussion, that’s my favourite thing about the books being successful: you get to talk about interesting things right in the middle of culture.”

The Last Devil to Die is no exception. It’s a crime story, yes. But at core it’s a book about dementia and assisted dying. Where his mother lives, residents are over 75 and, “They had a big debate about it, incredibly rational, incredibly polite. Lots of disagreement, [but] everyone listening to each other. People who have been medical professionals, people who’ve been mental health professionals and people who’ve obviously lost loved ones. It’s something that you’re allowed to talk about. It’s not crazy to want to die when you’re in pain with no way of getting out of that pain. I absolutely respect the views of people for whom [assisted dying] would be an impossibility. But it’s an argument that’s not going away. We have such control over our lives, it seems weird that the final bit we have no control over. An awful lot of people would sleep easier if they knew their last few years wouldn’t be very difficult.”

Osman watched dementia take possession of his working-class firebrand grandfather, watched him try to cling to moments of clarity. His mother told him that in hospital you could see his heart beating and knew it was never going to give up. “He was such a strong man. But he would absolutely not have wanted to be there.” Osman drew from this experience and also research. “The Alzheimer’s Society said, ‘If you’ve met one person with dementia, then you’ve met one person with dementia.’ That’s how I approached it really: knowing that everyone’s experience will be different.”

***

Richard Osman was born in 1970, the second child of David Osman and Brenda Wright. He grew up near Hayward’s Heath in West Sussex. He was born with nystagmus, an eye condition which means the world appears somewhat blurred. A building will appear like an impressionist painting, he has said, and he can’t make out the birds in the trees. Whether or not it’s a consequence, he is able to tune in to a multilayered soundscape: “If six different conversations are going on, I’ll hear every single one. So that’s a useful skill.”

When he was nine, his world was ruptured. He remembers going into the lounge with his glass of orange squash for a family meeting. There, his father informed him and Mat, who is three years older, that he’d fallen in love with someone else and was going away. He said he hoped that was all right. David Osman maintained six or so months of contact with his children before moving a distance that required them to take long coach journeys to visit. It was difficult for a child and Richard, in a tantrum, told his father he did not want to see him any more. His father took this at face value.

Brenda was left to take care of everything. “But God, if you’re brought up by one good parent, then you’ve hit the jackpot, haven’t you? So long as that kid knows they’re loved.” He adds as a mumbled aside that an awful lot of British politics can be explained by the lack of even one loving parent. “Because if you weren’t loved, what are you looking for? What you want to achieve is very different from someone who was brought up in a loving home.”

That said, he has no idea how Brenda managed. “It must have been hell. The fact that I didn’t know it was hell is a product of, a) me being a boy and not in tune, and b) she wore it so lightly. It was not something she wanted to trouble us with.” She did all the work that she could find in addition to her day job as a teacher, including stuffing envelopes for a fraction of a penny a go. Occasionally, he’d hear her weeping over the lost promise of her life. But as a child it was too difficult to take onboard, and he walled off those sounds of disappointment, disappearing instead into the world of TV. His eyesight meant he had to sit close, but television showed him the world as he couldn’t see it – birds in the trees, buildings. Sports action replays meant he could see the ball. He watched everything; mainlined British culture through the medium of television. He had idiosyncratic hobbies, such as making a World Cup of music bands (a format he later reproduced on Twitter, when he did the World Cup of crisps and Christmas sweets).

Meanwhile, Mat was in his room playing music. His mother let them be. “She gave us absolute educational freedom. Never made us go to piano classes or do French tuition. Never made us do homework. She didn’t push us to go to university, although I think she would have been horrified if we hadn’t. She played it cool. She just trusted us both. That’s an amazing thing to give to kids. You can’t really tell either of us anything – even now,” he laughs. “She thought, ‘These two seem to have an idea of what they want to do.’ That was her parenting style.”

I’d buy this more entirely if it weren’t for the faintly barbed quote Brenda gave to the Times about his writing style being “quite staccato”, which suggests she has no qualms chivvying her son. Either way, it paid off. Osman says he was probably the first from his school to go to Cambridge, where he did sociology and politics (although he still regrets not doing American studies at Leeds); his brother, “who is proper clever”, did economics at the London School of Economics.

On his father, who died in 2016, he shrugs. Even before David left, Richard doesn’t remember him being around a lot. He was a teacher, then something else, he’s not sure what. “Listen, he was fundamentally a perfectly nice human being. I don’t sit here trembling with fear in my heart when I think about my dad. I’d love to feel something, some stirring of emotion. I probably did in my 20s when I made contact with him again. I think I manufactured some anger.” Ultimately, “I don’t think parenthood was for him. So, he was able to divest himself of those responsibilities.”

While Osman had girlfriends in his teens, he didn’t really drink and wasn’t very “worldly”: “I didn’t really understand what people did. Throughout life I’ve picked up clues as to what it is to be human. I know what it is to be me. But to be other people? I always find it extraordinary.” He says his height and eyesight meant he kept to the periphery – he wasn’t going on big clique holidays or setting up theatre groups. He found sitting and listening “sociologically interesting”. After a beat, he says: “I blame a lot of things in my life on my eyesight, but I think actually it might just be my personality.”

He married young and had two kids, Ruby at 27, Sonny two years later. He will say precisely zero about his divorce: “It’s not my story, if you know what I mean.” He quickly adds that he’s hiding “nothing grim; it’s just her privacy”.

Whether work was going well, or his marriage was going badly, there was another issue nagging away. In his 30s he had an anagnorisis of sorts: went to therapy and was diagnosed with addictive behaviours, the most explicit around food. For years he’d suffered bouts of binge eating, “an absolute compulsion to eat, an inability to stop eating, shame afterwards and then repeat”. The pattern could continue for weeks or months. “I find myself in situations sometimes where my behaviour around food is so absurd, it makes no sense. It’s certainly not self-care.”

Does he remember what age he was when it started? “Oh, like 10 years old. Yeah, I wonder what the inciting incident was. And food when you’re 10 is something that you can’t control. You’re not going to become an alcoholic or a drug addict.”

While it doesn’t have the “doomed glamour” of alcohol or drugs, he has said, the behaviour is in essence the same – although “slightly more behavioural and slightly less to do with the substance itself”, as with love or sex addiction. “But the second you go to therapy, you realise that’s just a symptom of the problem. You realise you’re just numbing whatever pain; you’re numbing the things you don’t want to think or talk about.”

These are not things you can give up, he says, so you are faced with the lifelong challenge of controlling it. “I’m either controlling it or not controlling it at any given time.” He also believes an inability to control eating “is so insanely common. You can’t look around at the shape we’re all in and not think that there’s an issue. It’s not spoken about, because it’s still laughed at.” He no longer feels any shame associated with it, and the affliction has given him compassion. “When I think about how other people behave, I go, ‘Yeah, I get it. You should see what I do.’” He adds: “It’s impossible to be a human being and not have issues.” (Later he says in passing that he thinks Boris Johnson has food issues. “You can see that. He’s got everything issues. There’s nothing that will fill that hole.”)

He’s described his 40s as “really good fun.” He was single for much of it and there was a merry-go-round of dates. “I was always looking for the one, always knew I wanted to get married, absolutely wanted to fall in love. And, listen, I enjoyed the process. Friends would go, ‘I don’t think that is what you’re looking for. I think you enjoy playing the field.’ I would always say, ‘It really is what I’m looking for.’”

Were they chucking women at him? “Um, not literally. I had my 20s in my 40s, for sure. My 50s are my 30s. I settled with the love of my life. It’s an enormous stroke of good luck, but as soon as I met Ingrid, I thought, ‘Great. Done.’ I knew it.”

They met when she appeared on his show House of Games in summer 2020, and she moved in that October. The following year, he bought a ring and planned his proposal – which was to be in a special restaurant on the third day of a holiday in Oman. Once there, he got in an awful flap and blew the whole schedule by proposing on day one, tears all over his face. Is this an example of his inability to keep a secret? He laughs. “My heart wouldn’t let me. It was absolutely bursting.”

Teasing, I ask how he gets on with Oliver’s mother, Jo Gideon. Osman describes himself as a lefty and she is the Tory MP for Stoke-on-Trent, a red-waller instrumental in ousting Johnson. It’s the only time Osman looks terrified. He doesn’t lie. Instead he says: “Let’s not go there. We won’t go into that.”

• The Last Devil to Die by Richard Osman is published on 14 September (Viking, £22). To support the Guardian and the Observer buy a copy at guardianbookshop.com. Delivery charges may apply.