For years, crime fiction titles have topped the bestseller lists and library lending tables. That sales of the genre have now overtaken general fiction, as revealed at the London Book Fair last week, comes as no surprise to its readers, practitioners, critics and industry professionals.

We’ve always recognised its reach, dynamism, integrity and, increasingly, diversity. Yet its rise, and indeed acceptance, is still a mystery to some, with any number of narratives seeking to understand the phenomenon. This is about as helpful as trying to define exactly what makes a bestseller a bestseller, and, perhaps more important, how to spot the next big thing. At best it’s a lucky amalgamation; so many factors come into play. Being in the right place at the right time, with the right idea, and talent, might be one answer. But how the hell do you come up with the right idea?

There are reasons, however, why the crime genre is so effective as a fictional narrative form, often well away from the glitz of the charts, and why it translates brilliantly on to the screen, both small and large. And there is a growing critical sense as to why it’s so popular now. This has little to do with the old idea, now resurfacing, that in times of uncertainty we seek comfort, redemption and resolution; a place where good triumphs over evil and order is restored. Classic detective fiction, which came to the fore between the world wars, was built on those ideals, along with some very rigid and peculiar rules dreamed up by writers such as Ronald Knox and SS Van Dine.

As any serious writer knows, rules are there to be broken. No artist wants to be bound, or dictated to. This doesn’t mean that writers should ignore basic narrative concepts of purpose, pace and plot. Or, of course, character. I talk to my crime fiction students a lot about “menace and motivation”; simply, having characters want things, and then having things put in their way.

Readers, and viewers, like to be engaged, excited even. Writing something entertaining has an attraction, maybe missed by those who believe that if something is wildly popular it is less culturally worthy or intellectually challenging. Raymond Chandler, in his polemical 1950 essay The Simple Art of Murder, railed against Dorothy L Sayers, who suggested there was such a thing as a “literature of expression” and a “literature of escape”. His riposte was that “everything written with vitality expresses that vitality; there are no dull subjects only dull minds”.

Crime fiction has always been broad, with current parallels. A course we run at the University of East Anglia begins with a study of Agatha Christie’s Murder on the Orient Express and James M Cain’s The Postman Always Rings Twice, both from 1934. They couldn’t be more different. One is an absurd “locked room” whodunnit, the other a chilling first-person confession from someone awaiting execution. Effectively, crime fiction split, taking a tidy, restorative Christie, or unruly, apocalyptic Cain path.

Crime fiction in its broadest sense has always been hugely influential, particularly among so-called “literary” writers. William Faulkner compared Georges Simenon to Chekhov. WH Auden adored Christie and Sayers. André Gide was an admirer of Patricia Highsmith, Albert Camus based The Stranger on The Postman Always Rings Twice, while Eleanor Catton drew on Cain’s Double Indemnity for her Man Booker prize-winning The Luminaries. Martin Amis’s novella Night Train is a homage to Elmore Leonard.

John Banville (another Man Booker winner), who writes crime fiction under the pseudonym Benjamin Black, has said that the “modernist experiment is over”, and the literary novel is “in the doldrums”, whereas crime fiction reasserts the traditional literary values of “plot, character and dialogue”. Banville’s reception among the crime-writing fraternity got off to a difficult start when he suggested at a festival that he wrote his crime fiction quickly, and his literary fiction slowly, inadvertently implying that it was easier.

While the essence of writing crime fiction might come down to speed and fluency, crafting and control are vital. It’s not easy and few do it really well. A crime novel that works is as taut as a drum. Plus, readers can quickly sniff out a fraud – someone writing up or down, or for the money, and it’s now a very competitive market.

Approaching the right idea and themes seems to depend as much on prescience as talent. Gillian Flynn’s Gone Girl was instrumental in determining the popularity of what’s become known, thanks to Julia Crouch, as “domestic noir”. This is a development of the psychological thriller, which has its roots in early 19th-century gothic tales such as the satirical Northanger Abbey (interestingly reimagined by Val McDermid, one of this year’s Man Booker judges).

Gone Girl is every bit as dark and disturbing, and knowing – critically, socially and politically – as Flynn’s previous two novels, Dark Places and Sharp Objects. Paula Hawkins’s The Girl on the Train is also extraordinarily dark and disturbing, and adapted successfully on to the big screen.

Neither landmark books, nor their many imitators, are what you’d call comforting. One of their main appeals is the fact that they make you feel uncomfortable; that they invade your headspace, adding drama, fear and anxiety. They sweep you away from the everyday. They heighten your senses, and they surprise, not least because they are tackling a world we thought we knew intimately.

That crime fiction can still accommodate neat police procedural, dark psychological thriller and everything between, above and beyond (and there are some very interesting crime, horror and speculative fiction crossovers emerging), while continuing to develop and being ever more popular, suggests it’s the fictional form of our times.

It could still, however, look far wider. A new wave of talented and ethnically diverse crime writers includes Winnie M Li, Jacob Ross, Leye Adenle and Amer Anwar. Work from India, Nigeria, Singapore, South Korea and Brazil is gaining global attention. Film and TV adaptations might travel further and faster than the book, but the fundamental concept of crime drama is the same: life and death. Murder, as Chandler acknowledged in his essay, is serious business. It’s the telling, the understanding, that changes.

Henry Sutton is a senior lecturer in creative writing at UEA and director of the MA Crime Fiction. His novel, Time to Win, written under the pseudonym Harry Brett, is published in paperback on 26 April by Corsair