“I’m afraid I like strong contrasts,” Roald Dahl said, not long before his death in 1990. “I like villains to be terrible and good people to be very good.” Dahl himself gave a lie to that formulation. He is very easy to cast as a villain: even friends described him as bullying, overbearing, arrogant and impossible; he was a compulsive gambler, a distant and wayward husband, an unforgivable antisemite. But then, with the assistance of Quentin Blake, there are also the books. Tens of millions of children – myself included – fell under the spell of his joyful, wicked, silly, inventive imagination in stories that suggested he was not of the adult world at all, but still leader of a childhood gang. Books that initiated you, as he hoped and believed, into a lifetime of reading. Books that – despite the waning reputation of their creator – Netflix last year paid upwards of £500m for the rights to adapt.

Several biographers have attempted to fill the gap between these polarities. Jeremy Treglown’s 1994 book Roald Dahl: A Biography set the parameters. Unauthorised by Dahl’s second wife and children, but with access to many of his letters, and to his first wife, the actor Patricia Neal, Treglown offered a seductive analysis of the writer’s psychology. He argued that Dahl’s life of tragedy – the writer’s father and sister had died by the time he was four, he lost his own eldest daughter to measles at the age of seven, and nursed both Neal and their son Theo through severe brain injuries – gave him a profound emotional darkness and the desperate need to find ways to transcend it. Dahl had asked his daughter Ophelia to write an authorised book, but when that proved too tough an ask, the family asked Donald Sturrock, in 2010, to step in. Sturrock trod carefully around some of Dahl’s more uncomfortable behaviour and found a convincing fortitude and late-life generosity of spirit to balance it.

Matthew Dennison’s new life is a well-researched, compact book drawn from the existing record. There are no new revelations or notable interviews but it turns the baggy complications of Dahl’s life into something brisk and manageable. Dennison is alive to telling detail but pulls a few punches. The infamous 1983 interview that Dahl gave to the New Statesman, for example, in which he told Michael Coren: “There is a trait in the Jewish character that does provoke animosity, maybe it’s a kind of lack of generosity towards non-Jews” and: “Even a stinker like Hitler didn’t just pick on them for no reason” is referenced only in passing. Dennison tends to conclude that the charges against Dahl of lazy or deliberately provocative antisemitism are mostly evidence that contemporary “liberal shibboleths have little truck with former renown”.

He is good on the strangeness of the early life. Dahl’s father, Harald, had left Norway for Paris to be an artist. Somehow he ended up in south Wales, with a hugely successful business selling Norwegian timber as pit props to coalmines. After his death, Dahl, always preternaturally tall – he was 6ft 5in at the age of 15 – was forced to grow into a role as the man of the house. He craved a life of adventure, and discovered one, first as Shell’s representative in Dar es Salaam, then as a fighter pilot in the war (he crash-landed in Libya) and subsequently as a socialite (and spy for Winston Churchill) in Roosevelt’s Washington.

It seems justifiably telling to Dennison, though he stops short of admonition, that Dahl recast the story of his near-death experience in the war both in print and in anecdote to emphasise his own heroism. Having dragged himself from the burning wreckage, he was partly saved by a fellow pilot who landed beside him in the desert and held him through the night to keep him warm. By the time of Dahl’s autobiography Going Solo (1986) that part of the tale was lost from his accounts. He became the singular author of his own survival. This nascent megalomania appears to inform one sense of himself throughout his life.

The competitiveness expressed itself sexually before Dahl’s writing took off. In the US after the war the young fighter pilot was a magnet for wealthy married women of a certain age. One contemporary recalled that in that period: “I think he slept with everyone on the east and west coasts that had more than $50,000 a year.” Six months into his marriage to Neal, Dahl was convinced he should leave her: “I make the coffee in the morning,” he wrote to a friend. “She stays in bed. I work until lunchtime. Then I get my own lunch out of a can of soup.” He set about an attempted seduction of Gloria Vanderbilt, quickly abandoned. He and Neal went on to have five children, but it was never a happy union.

The life-changing books followed hard on the heels of tragedy. Charlie and the Chocolate Factory (Dahl had previously thought of children’s books as “an uneconomic diversion” from adult fiction and screenwriting) was completed in the year after seven-year-old Olivia died. Dahl had retreated into silence and drink; his daughter Tessa recalled how the family thereafter “toppled unwittingly over the edge of a jagged cliff face into a canyon of darkness filled with sadness, such total devastation that we would never recover”.

That despair was compounded by Neal’s brain haemorrhage during pregnancy in 1965 that left her in a coma, and requiring intensive speech and physical therapy for months and years afterwards. Dahl refused to accept the change in her, insisting on a fierce daily rota of rehabilitation, noting down the nonsense phrases she sometimes uttered for use in his books. Reading this account, you begin to recognise the appeal of his fictional heroes – Charlie and James and Matilda and Fantastic Mr Fox and all the rest – they are universally survivors against the odds, often orphaned, always alone against the world, before finding ways out.

In his own life, Dahl escaped into an affair with a family friend, Felicity “Liccy” Crosland, more than 20 years his junior. The affair lasted 10 years before it ended his marriage to Neal. He married Crosland at the age of 67 and entered in some ways the most productive phase of his life, certainly the most serene. Right to the end, however, he never lost that will to prevail against all comers. Not long before his death, Dennison notes, Dahl recalled having persistent, “champion of the world” dreams in which he won Wimbledon or Open golf championships. He often woke up thinking: “I’ve beaten them all and everyone’s surprised.”

• Teller of the Unexpected: The Life of Roald Dahl by Matthew Dennison is published by Head of Zeus (£20). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply