The American novelist John Barth claimed that rather than the traditional “what happened next?”, the real question that every reader is asking him or herself as they read is “the essential question of identity – the personal, professional, cultural, even species-specific ‘Who Am I?’” Stories are ordering, sense-making machines, helping our brains to render the frantic incoherence of chaotic existence into comprehensible narratives. These narratives, as Peter Brooks showed in his classic critical work Reading for the Plot, “follow the internal logic of the discourse of mortality” – stories have beginnings, middles and ends because our lives do. Every time we read a novel, we’re giving ourselves a new way of thinking about the shape and structure of our own lives. And even in the age of AI, the novel remains our most subtle and sophisticated piece of technology when it comes to answering these deep, existential questions.

It’s surprising, given how many authors now teach creative writing in order to supplement their meagre incomes, that there aren’t more good books on the craft of novel writing. Novice novelists still tend to turn to screenwriting guides when looking for inspiration. Yet as the brilliant send-up of Robert McKee’s Story, one of the many guides that use formalist archetypes to provide film writers with plot blueprints, in the Charlie Kaufman film Adaptation demonstrates, the structures that work for blockbusters don’t always work for more refined narratives.

Will Storr is a prize-winning journalist and the author of a very good, if largely forgotten, novel, The Hunger and the Howling of Killian Lone. In The Science of Storytelling, he attempts to do for novelists what McKee, Joseph Campbell and Christopher Booker did for screenwriters – providing a how-to guide that looks back to the fundamental questions that animate readers and then using these to help novelists shape their narratives. And yet Storr is doing something more interesting than merely cashing in on the current boom in creative writing and the bulk of this book isn’t only for those wishing to write themselves. Recognising that novels respond to deep psychological impulses, Storr employs a mixture of neuroscience and psychology to explore why it is that the novel has become such a staple of our cultural lives. He shows how novelists answer the challenge of “grabbing and keeping the attention of other people’s brains” by delving into the science of those brains.

This makes for a hugely compelling reading experience. Storr weaves brilliantly between high and low culture – in the space of a few pages we go from Mrs Dalloway to Gone Girl to Marion and Geoff to the computer game Fortnite – and he illustrates and expands upon these examples with repeated reference to the science that lies behind them. He makes particularly good use of the psychologist George Loewenstein’s work on curiosity (“deep in the detail of his dry, academic paper, Lowenstein has written a perfect description of police-procedural drama”) and the neuroscientific research of Benjamin Bergen and Michael Gazzaniga. Storr shows how novels activate the brain’s reward mechanisms; he illustrates how we draw on neural models to populate the worlds of the novels we’re reading; there’s some brilliant stuff on research into saccades – the miniature movements our eyes make when processing information – and how novelists might use this to structure their scenes.

The Science of Storytelling ends with a long, practical appendix, entitled The Sacred Flaw Approach, which offers a step-by-step guide to writing a novel, drawing on the lessons and observations of the book. Such templates usually feel a little blunt for the lengthy and complex art of novel writing, but Storr’s central thesis is so compelling, his own prose so well sculpted and readable, that I found myself largely convinced. Robert McKee has built an empire out of his screenwriting manuals – it’ll cost you close to $1,000 to attend one of his seminars. Storr’s superb exploration of the enduring appeal of the novel feels like it could do something similar – offering a smart, fascinating exploration of the science and psychology behind our most sophisticated art form that also works as an effective how-to guide.

• The Science of Storytelling by Will Storr is published by Willliam Collins (£12.99). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99