Neil Gaiman is relieved to be home. Dressed all in black, unruly black hair shot through with silver, he sits comfortably on a couch at his house in Woodstock, New York. “I left here in August 2019, figuring I’d be back by the end of March,” he says. “I didn’t actually get back till April 2022. Just in time to go off on a tour of America.” Nobody was completely unaffected by the pandemic, but not even Covid could derail the big-budget Netflix adaptation of Gaiman’s seminal 90s comic series The Sandman, due next month.

The British-born Gaiman is one of the most recognised living authors. His restless imagination refuses to be constrained by genre or demographic; he has written adult horror fantasy (American Gods), children’s literature (Coraline) and retellings of ancient myth (Norse Mythology). He has won the Hugo, Nebula and Bram Stoker awards, and the Newbery and Carnegie medals. And those are just the books. Gaiman’s Midas touch extends beyond literature: his play The Ocean at the End of the Lane was recently staged in London to enthusiastic reviews; film adaptations of his work include Coraline and Stardust; Lucifer, Good Omens and American Gods have all been successfully adapted to television; Gaiman even wrote the English translation of classic Studio Ghibli anime Princess Mononoke.

Gaiman’s output may be prolific and diverse, but it’s also instantly recognisable. It is characterised by a magic realist view of the universe, where mythic creatures exist within the everyday. There is also a kindness that pervades his stories; Gaiman is a writer who likes his characters. This generosity may be one of the reasons he enjoys such devotion from his fans. This essential optimism is also apparent in his social media interactions. He is unselfconsciously familiar with his almost 3 million Twitter followers and the readers of his online journal; fans were regularly invited to listen to Gaiman and his wife, the musician Amanda Palmer, singing lullabies to their son, Ash.

Gaiman may be enjoying phenomenal success now, but he started in comics, then considered the literary gutter. The mid-80s was an exciting time to be involved with the medium, and Gaiman – working as a freelance journalist – was well aware of it. “In 1986, I pitched a story about what was happening in comics,” he says. “At that point Maus, The Dark Knight Returns and Watchmen were all coming out. One newspaper replied: ‘We’ve written about Desperate Dan’s 50th anniversary this year – we can’t do another comics piece.’”

Gaiman was eventually commissioned by a Sunday newspaper supplement to do a feature on comics. “It would have been a big cover article. I did all the interviews and gathered together art, including unpublished material.” But it was not to be. “They didn’t call back,” continues Gaiman. “After a few days, I phoned the editor to ask if he got the article and he said: ‘Yes, I’ve read it. There’s just one problem – we feel it lacks balance. These comics – you seem to think they’re a good thing …’ What I was saying was that there was this whole flowering of the medium, and what he wanted was for me to interview people who felt that comics were the end of literacy.” Gaiman was, of course, proved right. Maus by Art Spiegelman went on to win the Pulitzer prize. The Dark Knight Returns was a bestseller that reached readers far beyond comics audiences. Watchmen by Alan Moore was named one of Time magazine’s top 100 novels since the magazine’s inception.

The late 80s saw a kind of “British invasion”, a migration of talent to the US that was to change the face of comics. This was largely due to Karen Berger, visionary editor of DC Comics, home to Batman and Superman. First, Berger commissioned writer Alan Moore to work on the horror title Swamp Thing. Moore’s run raised the bar for what was possible in a monthly comic. The door was open. The Scottish writer Grant Morrison followed. Taking on the lacklustre superhero team Doom Patrol, Morrison twisted it into their own surreal, psychedelic image. Now anything seemed possible.

This was the context for the largely untried Gaiman to enter the picture. The way DC comics worked was to offer these young British upstarts an unimportant existing character to play with, as a way of minimising risk. Gaiman was offered The Sandman, a minor character who had appeared in two incarnations over the years, neither of them very successful. Gaiman was encouraged to make the character entirely his own and was given free rein to make any changes he saw fit. Even with this tabula rasa, Gaiman was nervous.

“Bear in mind, at this point I’ve written and sold maybe four short stories and [comic miniseries] Black Orchid. And now I’m going to have to do a monthly comic,” he says. “And I have no idea whether or not I can do it. I don’t think I have the engine to write a superhero comic. I’ve watched what Alan Moore does, what Grant Morrison does. These guys have superhero engines, they can do them; I don’t have that.”

Gaiman needed another way in, and it came via a US science-fiction author. “Roger Zelazny did a book called Lord of Light, where he did science-fictional gods who feel like superheroes,” says Gaiman. “It’s set in a world in the future where a bunch of space explorers have given themselves the powers of the Hindu pantheon. I thought: I can’t do superheroes, but I could do god comics. I bet I could get that kind of feeling to happen, and it might feel enough like a superhero comic to fool people.”

Gaiman’s version of the Sandman is Morpheus, a handsome goth as comfortable in a flowing cape of velvety shadows as he is in stove pipe jeans and a T-shirt (all in black, of course). He’s one of the Endless, seven immortal siblings who are the embodiment of natural forces: Death, Desire, Destiny, Despair, Delirium, Destruction and – in Morpheus’s case – Dream. Like all mythological deities, Morpheus’s almost unlimited powers don’t protect him from heartbreak and jeopardy. In fact, Gaiman delighted in exploring the fallibility and essential humanity of these figures. Morpheus’s immortality also allowed Gaiman to set his stories in every era, from deepest prehistory to the far future, as well as providing a snapshot of modern culture. It was an audacious concept for a superhero series.

With this magic equation in place, Gaiman’s “god comic” took off. Rich with literary allusions and foregrounding strong female characters, The Sandman found audiences outside the usual legion of fanboys. It became essential reading on college campuses – for both male and female students. “Back then, DC always did a year before they cancelled a title,” says Gaiman, “so I figured at issue eight they’d phone me up and say: ‘Well, minor critical success, major commercial failure. You’ve got four issues to wrap it up!’ Then I’d be done. That would be The Sandman. Instead, we got to issue eight, and we were selling more than anything comparable had ever sold within the last 15 years.” His confidence buoyed up by the sales, Gaiman was determined to do precisely what he wanted with the comic, supported by his ally Berger. “I knew this was the only chance I was ever going to get to put all the stuff that I loved in a comic,” he says, “so if I wanted to do a retelling of [Roman historian] Suetonius’s Life of Augustus as a homage to the poet Robert Graves, I was only going to get to do that once. The luckiest thing I had was an editor who trusted me.”

Sales climbed – particularly when the comics were collected in paperback book form and made available outside of comics shops – but critical acclaim was slower to follow. “In 1989, I was at a Christmas party for some magazine that was at the Groucho Club,” says Gaiman. “I got talking to somebody who, if memory serves, was literary editor at the Telegraph. They asked what I did, and I said I wrote comics. The gentleman looked as though I’d slapped him with a herring, but he couldn’t just stop talking to me and turn away, so he asked: ‘Oh well, what kind of comics?’ I said I’d written a thing called Violent Cases and that I was writing a thing called The Sandman. He said: ‘Hang on, you’re Neil Gaiman?!’ I said: ‘Yes.’ He said: ‘Oh my dear fellow, you don’t write comics, you write graphic novels!’ I felt like a hooker who’d just been called ‘a lady of the evening’.”

A successful writer can enjoy a whole career without creating a classic. Gaiman created his right out of the gate. To those who read it, The Sandman was as much a part of the 90s as Twin Peaks and Nirvana. It may not have initially had audiences on the same scale, but it created an icon of its writer. “In 1997, I went for a meeting at Warner Bros. I don’t even remember what it was about,” says Gaiman. “I’m with my new agent. The meeting doesn’t go anywhere, and when we get downstairs there are a couple of what would then have been called secretaries at their desks. As we pass one of them says: ‘Excuse me Mr Gaiman – would you sign my Sandman?’, which I do. My agent says to me in a jocular fashion: ‘Ha! Ha! You’re like Tom Cruise to these women.’ She turns to him furiously and says: ‘Tom Cruise and Brad Pitt are in here all the fucking time. This is Neil Gaiman. He wrote Sandman!’ What’s fascinating to me about that is that nobody at the studio back then knew who I was, but the bottom-level people, the ones who brought the bottle of water when you came, did. Those people are now running the studios. They’re the ones determined that the Sandman they loved gets on to the screen.”

The TV adaptation is sumptuously presented and faithful to the original comics – but not slavishly so. One of the reasons The Sandman initially found such a passionate audience was its progressive views on gender, sexuality and diversity, at a time when those views weren’t popular, let alone de rigueur. That has made it easier to adapt. “The great thing about having done all that stuff back then is that there’s an awful lot less work to do now,” says Gaiman. “There are moments when people yell at me for being woke online, and I’m like: ‘Have you ever read the fucking comic?’ People have criticised me for casting a gender-fluid, non-binary actor as Desire, but they were in the original. Desire was always non-binary; that was the whole point of the character.”

These issues weren’t exactly at the top of the agenda when The Sandman was first published. “In the early 90s I was getting flak – particularly from the British comics press – for not being political enough. People were doing political comics about how Margaret Thatcher was a bad person, and I was talking about diversity of race and gender and sexuality. When I was asked about it in interviews I would say: ‘Well the personal is political, and maybe you’ll find that Sandman is more political than you think.’ I wasn’t out there banging a drum. These were my friends and I wanted to put them into comics. I wanted to change hearts and minds. I had trans friends and still do, and it seemed to me that no one was putting trans characters in comics. And I had a comic. I would learn things from my friends: for instance, people being buried under dead names [the names they used before transitioning] by their families. That was shocking to me. And I thought, here was this opportunity to write a story in which everybody reading it is going to fall in love with my trans character who is going to be awesome.”

The Sandman adaptation has a stellar cast, including Charles Dance, David Thewlis, Joely Richardson, Stephen Fry and Mark Hamill. The title role of the Sandman – AKA Morpheus – is played by Tom Sturridge. As well as being the story’s creator, Gaiman is executive producer and a writer on the show. Casting was important to him, particularly the lead. “The lines are very difficult to say. And he does it. He has the gravitas. You feel for him. He can be a dick and you still love him. It’s magic.”

Game of Thrones’s Gwendoline Christie plays Lucifer, the ruler of hell. She’s scene-stealing good, obviously relishing her wicked role. Gaiman concurs. “We needed someone gender fluid. As gender fluid as the young David Bowie when he was a folk singer; he was my original model for Lucifer. We found Gwendoline Christie. And she’s enormous! And she’s beautiful! She has that power!” Then there’s Morpheus’s sister, Death, played by Kirby Howell-Baptiste. The chemistry between the siblings is one of the key turbines of the story. “Casting Death was horrible,” admits Gaiman. “I saw 700 auditions. It’s amazing how many amazing, beautiful, talented actresses of all ethnicities and backgrounds can’t say lines like: ‘You are the most appalling excuse for an anthropomorphic personification on this or any other plane!’ and make it convincing.”

With The Sandman finally birthed, Gaiman is already turning his attention to a broad slate of projects, some literary, some for television, some for the big screen. He is also returning to his roots by completing his famously unfinished comic series Miracleman with artist Mark Buckingham for Marvel comics. “I’m primarily a storyteller,” he says. “As a kid I really wanted to write comics. Comics were fabulous. They were a place you could do things that nobody had ever done before. But I also wanted to make movies. I wanted to make television. I wanted to write novels. What I wanted to do was tell stories.”

Neil Gaiman on Sandman’s visual inspirations

Original artwork for Sandman Issue 1, by Dave McKean

It was about three feet high and had shelves down the side. People assume that it’s a collage but it’s not; it’s an object with a painting in the middle and all sorts of found objects along the side. Dave had incredibly high standards. We were so lucky to have found each other at the time we did. He hated superhero comics; he hated pulp; so I knew my first reader was someone I was going to have to impress.

The Beguiling of Merlin by Edward Burne-Jones

I wanted Sandman to feel pre-Rapahelite. It was all about the build. Barry Windsor Smith did a thing where he took pre-Raphaelites and added muscles to create his comic Conan the Barbarian. What I did was just strip off those muscles.

Metal Machine Music by Lou Reed

When I wrote Hell [a location in The Sandman Vol 1], I played that album for four days solidly. I don’t know what the soundtrack to hell is, but Metal Machine Music will do. It’s still unlistenable, but when I hear it I’m incredibly happy because that was my soundtrack.

King of America by Elvis Costello

Sleep of the Just was on this album. After writing Sandman issue one, I remember not being able to come up with a title for it. I found myself humming something and I thought: what is that? I know that song. I realised I was singing Sleep of the Just and thought: Oh, that’s the title!

Sandman’s Helmet from the Netflix show

The first DC comics character to bear the name The Sandman wore a gas mask. I wanted a visual tribute to that gas mask, and I thought: let’s make it the kind of thing HR Giger would have created. The idea was a metal, leather and bone gas mask, essentially made out of a spine. I loved the fact that different artists would draw it differently. It only became a problem when we had to make one for the show, because there were so many versions! We eventually picked the one JH Williams III had drawn in The Sandman: Overture. That’s our helmet.

• This article was amended on 23 August 2022 to correct some personal information.

The Sandman is available on Netflix from 5 August.