Ever since 1950, when the BBC gave us the live, first-ever broadcast of A Christmas Carol, we've relied on Charles Dickens to see us through the winter months, as we draw the curtains, stoke the fire, and sit down in front of the box with a hankie, ready to weep and chuckle our way back into his vivid, awful, exciting 19th century, when people had wonderful names like Scrooge and Cratchit, and every sentence was longer than this one.
We are now in Dickens Year, it being the 200th anniversary of his birth on 7 February. By way of celebration, the BBC has already given us a new Great Expectations, starring Ray Winstone and Gillian Anderson. And following on from that old favourite, there's an invitation to enter a darker, stranger world, a shadowy place of drug addiction, illicit lust and murder. Gentle reader, welcome to my world.
My pitch to the BBC to complete the author's great unfinished novel was short and sweet: "The Mystery of Edwin Drood. Episode one – by Charles Dickens. Episode two – by moi!" This made some important people laugh and got me the commission. As a humble crime writer, I was thrilled. My first classic adaptation!
It's fairly well known that Dickens died halfway through writing his murky story about an opium-addicted, erotically obsessed choirmaster called John Jasper, who plots to murder his nephew and love rival, Edwin Drood. What's less well known is that Dickens died on purpose – to avoid having to finish it. Or that's what I came to believe, after months of wrestling in darkened rooms with the questions he ran out of time to answer. Who actually kills Edwin Drood? What is the meaning of that strange and awful cry in the night? What brings the prickly and defensive Landless twins all the way back from Ceylon to the sleepy fictional cathedral city of Cloisterham? And is that really a big white wig on Mr Datchery's head?
As always, the writing in Dickens is so magisterially confident that, on first reading, you are absolutely sure the author knew where he was going. After all, in every corner lurk what must surely be expertly placed clues: Jasper's black silk scarf, seemingly full of murderous intent; Mayor Sapsea's tomb and its enormous key; that pile of quicklime crying out, "Notice me!" But the more I studied clues in the text, and others I came across in conversations between Dickens, his friends and his family, the more the whole enterprise rocked like a demented house of cards. Some clues lead nowhere. Some are contradictory. Some just plain wrong. That pile of quicklime? Dickens seems to have believed it dissolved human flesh. Did he intend to pop a dead body into it, one that would later be identified by an undissolved ruby ring? Perhaps – but quicklime actually has a preserving effect.
In any case, there was more to finishing the story than solving a murder mystery. Drood, a tantalising network of puzzles, was intended to be a much shorter novel than was normal for Dickens. Only 12 monthly instalments were planned, rather than his usual 20, the intention being to write something exciting, suspenseful and tight; he would have loved the term thriller, but it had yet to be invented.
Even his fans, of which I'm one, would have to admit that he lacked one or two core requirements of the genre, though. Plotting, for instance. Dickens loved observation and digression and wonderful oxbow lakes of inspired daftness. But thrillers get confused by all that. Thrillers love plots. They love twists and turns and surprises. They want everything that happens to be significant, so that at the end the audience has the satisfaction of reaching the answer at exactly the same time as the storyteller.
Whereas Dickens' endings, well, just think of Little Dorrit, where someone suddenly and conveniently turns up from Russia and saves the day with a big fat cheque. So, even as I began to think I could discern his intentions, the relief came tinged by the dawning suspicion that what he intended was unlikely to satisfy a modern TV audience. I also began to suspect that Dickens himself had sensed his planned ending was flawed – and might even have meant to change it.
At this terrifying moment, with a chasm of hubris opening beneath my feet, his wise and sensible favourite daughter came to my rescue. Katey Dickens Collins, I discovered, had observed that her father's brain was clearer and brighter than usual during the writing of Drood; and, intriguingly, she did not think the murder mystery was what most appealed to him about the tale. Instead, she directed readers to his "wonderful observation of character, and his strange insight into the tragic secrets of the human heart".
So I stopped trying to channel the intentions of a dead man and turned instead to his great creation: John Jasper, the living, breathing, unforgettable character at the heart of this wonderful story. Instead of asking myself what Dickens wanted, I asked what Jasper wanted: lonely, raging, unloved Jasper, one of the most compelling and heartbreaking antiheroes in fiction. Naturally, I'm not going to reveal whodunnit in my ending here – but I will say the decision to complete the story from Jasper's point of view helped me answer the first question facing every screenwriter adapting a novel: what will I leave out? To those who love the book, I apologise for the loss of the trains, the weir, former seaman Mr Tartar, widowed landlady Mrs Billickin, bullying philanthropist Mr Honeythunder – and, in particular, the flying waiter.
Doctor Who does Drood
Left with the characters closest to the mystery, I gently pushed them in new directions. I gave wide-eyed little Rosa Bud, the 17-year-old object of Jasper and Drood's affections, something more to do than suck pear drops. I came up with eight possible identities for the mysterious newcomer Dick Datchery, before settling on the one I found most amusing. And I fell enough in love with the bright, cheerful, lonely Reverend Crisparkle to want to give him a happy ending.
Helena Landless and her fiery brother Neville presented a visual challenge. These young orphan twins are from Ceylon, but have English names. I inspected the original illustrations closely. Were those crisscrossing lines on their faces meant to suggest brown skin? How exciting! I decided, on no textual evidence, that they had a British father and a Tamil mother. With great enthusiasm, the production team put two young British Asian actors into starring roles in a costume drama for the first time.
Doctor Who, in a 2005 Christmas special, suggested Drood died at the hands of alien beings called Blue Elementals. A 1980s American musical version, which hit London's West End and starred Lulu and Ernie Wise, ended with the audience voting on which of several endings they preferred: every night a different ending. But from the moment I gave my heart to John Jasper, I knew exactly what my dark hero's final scene had to be. Of course, if I embarked on it all again next year, I might solve the puzzle differently. But that's all part of the charm of Dickens and Edwin Drood – his last present to the nation.
• The Mystery of Edwin Drood is on BBC2 on 10 and 11 January
