Three of the key roles in the career of Benedict Cumberbatch have been a hyper-intelligent drug addict (in Sherlock), an arrogant toff (in Parade’s End) and a young man deranged by thoughts of his dead father (in Hamlet). His latest TV part feels like an extraordinary combination of this trio. Cumberbatch plays Patrick Melrose in an adaptation of Edward St Aubyn’s quintet of autobiographical novels about a young man whose sexually abusive father and emotionally damaging mother lead to a heroin, cocaine and alcohol dependency that is treated first by rehab and then a writing career. The opening episode is a supreme piece of acting: a 60-minute near-monologue of craving, raving, shaking and sarcasm – occasionally interrupted by immigration officials, drug dealers, undertakers and relatives – as Melrose, in early 1980s New York, tries to collect his dead father’s ashes from a funeral parlour. Has any actor in TV history ever spoken at such speed and yet with such clarity for so long? Each episode (dramatising a single novel) is designed to have a distinct feel: the second is languid and ominous, set in Provence during the slow nightmare of Patrick’s childhood.

The first two Melrose novels – Never Mind and Bad News – were published in 1992, with the sequence completed by Some Hope (1994), Mother’s Milk (2006) and At Last (2011). Although the story has taken more than a quarter of a century to reach TV, there have been previous attempts. “I think all five novels have been permanently optioned by some company or other over the last 26 years,” St Aubyn says. “But you need so many things to come together – casting, budget, director, timings – that it begins to seem incredible that anything ever gets made at all.” One problem in convincing broadcasters to commit, the novelist admits, is the nature of the protagonist. A representative paragraph of Bad News runs: “He had injected whisky, watching his burned vein turn black under the skin, just to satisfy the needle fever. He had dissolved cocaine in Perrier because the tap was too far for his imperious desire … He had woken up after passing out for thirty hours, the syringe, still half full of smack, hanging loosely from his arm …”

The block to adaptation was eventually broken when the novels were published in the US in 2012. “Before that,” St Aubyn recalls, “Americans wouldn’t touch them because they were ‘too English’ or ‘too French’ or whatever.” He chose a US-UK production team of Rachael Horovitz, whose shows include HBO’s Grey Gardens, and Michael Jackson, a former channel controller at the BBC. When Cumberbatch mentioned in an online Q&A that the role he most wanted to play was Patrick Melrose, they had their star. Novelist David Nicholls (Starter for Ten, One Day) had been obsessed with Melrose since picking up Never Mind in 1992, while working in a bookshop, so quickly signed on to write the screenplay.

Both St Aubyn and Nicholls see aspects of Hamlet in Melrose and so felt it to be a good omen when Cumberbatch was a success in the 2015 Barbican stage production. In Bad News, Patrick wonders “How could he think his way out of the problem when the problem was the way he thought”, which is a Melrosian spin on “to be or not to be?” Strikingly, Cumberbatch recently said that “Hamlet and Melrose are the only two roles I’ve ever bucket-listed.”

The attitude of novelists towards screen adaptations runs from taking the money and weeping to having, like JK Rowling, a full creative veto. Where is St Aubyn on that spectrum? “Well, I certainly haven’t been weeping and, actually, the money isn’t huge. People always imagine it’s more than it is. And I sought no control over the outcome.” The author is credited as a “consultant”, a role that mainly involved reading and commenting on Nicholls’s scripts.

Initially planned as four 90-minute episodes, the series became five one-hours but after much discussion the order of the first and second novels was flipped, largely because Patrick is a child (played by Sebastian Maltz) in Never Mind, in which the brutal sexual abuse is revealed. “Cumberbatch has many talents but he can’t play a five-year-old boy,” St Aubyn remarks. “I think, in narrative terms, it works very well that viewers see Patrick as he is, and then what happened to him in the past.”

Jessica Raine, who plays Patrick’s ex-girlfriend Julia, had read all the books several years ago: “I would recommend them to friends, some of whom would have very different reactions to me; they found the world too dark.” Jennifer Jason Leigh plays Patrick’s mother and the Australian actor Hugo Weaving his father. St Aubyn, who makes a cameo appearance in the third episode at a party for Princess Margaret, was often present on set, and happy to discuss the characters with the actors. “He immediately said: ‘Let me tell you about my father, and what happened to me’, ” Weaving says. “Even though I’m playing David Melrose, not St Aubyn’s father, it was good to have some sense of the man. It was a hard role to play, but he liberated me to do it without worrying about ‘the truth’ or upsetting him.”

While acknowledging that the “central psychological story of the novels is true”, St Aubyn is keen to stress that they are necessarily far from being a disguised transcription of his life: “Patrick is five in the first book. Obviously, at that age, I wasn’t sitting in the corner of my parents’ dining room like a little Boswell. I made it all up, and David and Eleanor Melrose are fictional characters.”

Screenwriters often complain that TV drama commissioners seek to sanitise dark material, encouraging characters to be more sympathetic. But Patrick Melrose looks and feels strikingly grownup, and Nicholls confirms that “there was no pressure to make the characters more attractive or ingratiating”. He was impressed that Cumberbatch “was quite prepared to be dislikable, self-pitying – and not to soften any of that. He shows you that Patrick’s manner is part of the damage inflicted by his upbringing. I think the usual use of wealthy people in TV drama is for aspirational, picturesque wish fulfilment – the houses, the clothes. But this show is really tough about what Patrick actually calls ‘these terrible people’.”

Watching the series, it struck me that St Aubyn now had access to three different versions of some people and scenes: his memory, then the fictionalisation and now the dramatisation. Was that upsetting or strange psychologically?

“It would be entirely reasonable to think that I might end up with levels of lived experience, and then the book and then the TV series, and that I would be in a snowfall of confusion about what happened when. But that isn’t how it was.” He points out that the French house in the novels was based on his grandmother’s, rather than his parents’, and the chateau in the TV series is different again, so the page and then the screen versions get progressively further from any triggering incident.

“I actually think it might have been a problem if Never Mind had been filmed soon after 1992. But, by this point, the novels are a sublimation of what happened, and the TV series is a double sublimation. Clearly, I wasn’t watching the story as an innocent viewer but, as it went on, I was surprised by the extent to which I could almost do that.”

• Patrick Melrose starts on Sky Atlantic on 13 May at 9pm, and will also be available through Now TV.