Will Self greets me on his doorstep, trailing a small child and an even smaller dog. It could be, of course, that their smallness is accentuated by Self's impressive height: at 6ft 5in, he moves with the languid grace of a man accustomed to folding himself into constrained spaces. "Oh, sorry, you were meant to have been let in," he says in a lugubrious monotone, clear blue eyes flicking this way and that. He speaks without smiling and yet you get the sense that he is perpetually on the brink of making some brilliant joke, comprehensible only to himself. Self opens the front door and ushers me inside. "Go on up," he says, nodding towards the staircase. "You'll know which room is mine."

There are three storeys to the tastefully decorated Georgian townhouse in Stockwell, south London that Self shares with his wife, the journalist Deborah Orr, and their children – Ivan, 14, and Luther, 10. The walls are painted in muted aubergines and greys; books line the shelves and everything is orderly without being twee.

Self's study, by contrast, is a resolutely untidy attic room exploding with paper. The walls are smothered with a complex mosaic of overlapping yellow Post-It notes. Every spare inch is covered with Blu-Tacked scraps: drawings by Self's children, images of unknown Edwardians taken from Ancestry.com, and – in one easily overlooked corner – an advertisement for sanitary towels from a turn-of-the-century magazine.

It is, perhaps, not surprising that Self should need to work in a state of such organised chaos. His latest novel, Umbrella, is a dazzling feat of imagination and structure: a sprawling, lyrical, stream-of-consciousness narrative that squares up to modernism and brings it kicking and screaming into the 21st century (the title is taken from a James Joyce quotation: "A brother is as easily forgotten as an umbrella").

The book is stomach-lurchingly ambitious in scope, spanning 92 years and examining the social legacy of the first world war, anti-psychiatry, the relativity of madness and the impact of technology on the human body.

There are three interwoven strands. In 1918 Audrey Dearth, a munitions worker, is incarcerated at Friern hospital after falling victim to encephalitis lethargica, a brain disease that swept through Europe in the aftermath of the first world war and left some of its victims speechless and motionless. In 1971 she is treated by psychiatrist Dr Zack Busner, who wakes her from her stupor with a new drug. By 2010 the asylum has been turned into a luxury apartment complex and Busner travels waywardly across north London in search of the truth about his past encounter with his former patient.

There is no straightforward chronological narrative: the three time zones are spliced together in intriguing, interconnected ways, sometimes within the space of a single paragraph. It is a portrait of an age but also of a city. Just as Joyce revealed the sprawling subtlety of Dublin in Ulysses, so Self undresses London, layer by layer, in all its pulsating complexity.

No wonder he needed all those Post-It notes.

"I don't really write for readers," Self says when he appears, bearing a packet of coffee. He lights a compact gas camping stove on the corner of his desk and puts a stainless steel espresso maker on to boil. "I think that's the defining characteristic of being serious as a writer. I mean, I've said in the past I write for myself. That's probably some kind of insane egotism but I actually think that's the only way to proceed – to write what you think you have to write. I write desperately trying to keep myself amused or engaged in what I'm doing and in the world. And if people like it, great, and if they don't like it, well, that's that – what can you do? You can't go round and hold a gun to their head."

As it happens, Self is pleased to discover I did like the book. "It makes a difference," he says. "My publishers did look a bit grim when it came in. I think they felt it was resolutely uncommercial and wouldn't find readers. But you know…"

He drifts off. Five days after we meet, Umbrella is longlisted for the Man Booker prize. It turns out some other people must like it too.

At 50, Self has carved out a reputation for himself as one of the UK's most uncompromising and interesting novelists. He was still in his 20s when he published his first book, a collection of short stories, under the title The Quantity Theory of Insanity. The book won praise from Doris Lessing and Salman Rushdie and scooped the Geoffrey Faber Memorial prize in 1993.

"It was published, as Cocteau said, to a terrifying baptism of caresses," Self recalls. "I didn't get a single negative review. For the one year, between the publication of the first book and the publication of the second book, I couldn't put a foot wrong."

Since 1991 there have been six subsequent collections of short stories, seven compilations of Self's non-fiction writing, one illustrated novella and eight more novels (Umbrella is his ninth) as well as a slew of journalistic assignments. Earlier this year he became the professor of contemporary thought at Brunel University and has just written an undergraduate module for the course.

Does he ever get writer's block?

"No. I get what I call 'everythingitis'… where I get obsessed with the idea that everything has to be in the book."

His novels defy easy categorisation and one senses it is partly for this reason that he has never fully been embraced by the reading public or attained the popular heights of his near contemporary, Martin Amis. In My Idea of Fun (1993), ostensibly a story about a lonely boy growing up in a caravan park, Self created a sinister alternate universe peopled by menacing characters from children's jokes, including a grotesque version of The Fat Controller.

In How the Dead Live (2000) he charted the afterlife of Lily Bloom, an elderly woman who was moved to live in a London suburb after her death, accompanied by an Aboriginal spirit guide. The Book of Dave, published six years later, was the story of a London cab driver in the throes of a mental breakdown who wrote a book of rantings that was rediscovered after 500 years and used as the sacred foundation for a new religion.

If there is a connecting thread in his fiction, it is, perhaps, a capacity for antic experimentation and a desire – even a compulsion – not to be boring.

"I have no patience with naturalistic fiction, really, I just find it dull," says Self. He cites JG Ballard as his most formative influence and confesses he doesn't read modern novels, partly because he's worried that if he reads something exceptional "it will fuck with my head and I'll get discouraged".

And yet he seems exhausted by his own single-mindedness. "I really don't say this for effect or because it's false modesty, but every book I write feels like a failure to me," he says, "and the failures feel worse as I get older. And I don't even like it in myself that I'm not more catholic [in taste].

"It's only really in the last decade or so that I've started to engage seriously with what I think the implications of modernism are in terms of the novel, and I wish I didn't have to, frankly, [because] I don't write to get readers but you can more or less guarantee that you'll start shedding them at that point." He emits a low, rich, rumbling laugh. "And I feel isolated enough as it is. Everybody loves, you know, what's-his-face, beardy guy…"

Alan Hollinghurst?

"Alan Hollinghurst!" he cries, gratefully. "Everybody loves Alan's books. Tout le monde. And I can't read them, so… Everyone goes: 'Ohhh, The Line of Beauty,'" he says, doing an excellent impression of a literary luvvie. "You'd be inclined to think, if you pick up The Line of Beauty and think: 'OK, so I don't actually need to go on with this', [that] the deficiency lay in you. You'd have to be very courageous to think: 'They're all wrong, this is not where the party is.'"

For Self, the party is definitely elsewhere. It's just that his party might not sound much fun to someone in search of an engrossing beach read. Umbrella, after all, is predicated on the modernist belief that it is only in the rejection of conventional linear structure and unity of plot that essential truth will be found.

But it is hard not to admire Self's determination to stick to his principles. "You can't go on pretending that the writer is an invisible deity who moves around characters in the simple past," he says. "I just can't do that stuff. It's lies. The world isn't like that any more. The world is really strange. It's not to be explained by 'He went to the pub'. You cannot capture what's going on with that form, to my way of thinking. You can create a divertissement, you can create a very fine entertainment, but you can't reach any closer to any kind of truth about what it is to exist."

The coffee pot boils, no doubt in a deliberate attempt to impose dull, naturalistic order on the contemplation of what it means to exist. Self unfurls himself from his ergonomic computer stool and gingerly removes the pot from the stove using a grubby towel.

"It's a great privilege to be allowed to have a filthy garret room," he says as he pours me a deliciously strong cup of black coffee. (Later he will tell me he uses a Robusta "peasant" blend bought from his local newsagents, which tastes smoother than the more expensive Arabica we're used to drinking. Self's conversation is full of such interesting digressions, the product of a restless mind accumulating facts like magpies do glitter.)

Usually the window above his desk overlooks the street outside, affording Self an uninterrupted view of the local crack dealers. Today, however, the entire facade of the house is obscured by scaffolding – the result of his roof dramatically collapsing without warning in May after a sudden rise in temperature.

"I was in the front room downstairs looking out the window. The boys and I were at home and it was like – do you remember those old Ray Harryhausen films where the special effects are quite crap, there's a definite sense of planes shifting against each other? – it was like that. All this masonry and brick just went past the window, with great rumbling and dust clouding."

Self got his children out on to the street and no one was hurt. "If Deborah had been working in the front garden all day, she'd have been dead."

Death and illness have been much on Self's mind of late. Orr was diagnosed with breast cancer in June 2010 and underwent a mastectomy in August, followed by a gruelling course of chemotherapy. Self doesn't like talking about this – there is an occlusion in the eyes, a quietness when the topic is broached.

"I think she's all right," he says. "I mean, you'd have to ask her really, wouldn't you? She's not technically in remission yet but she's not got cancer again so we'll see."

It is clear, although he never explicitly states it, that he adores Orr. One of the great achievements of Umbrella is Self's ability to write a convincing female protagonist. Unlike many of the great male novelists of his generation who have been accused of ignoring the female experience in their work, Self seems entirely at ease with the female psyche.

"Yeah, well, I'm quite girly," he says. "I like women, my friends are women. I don't have many male friends – never have.

"I always start with physicality when I'm writing as a woman. So I always have a vagina and think about having periods. I always start with an embodiment. And I think when I read men writing about women, they never seem to have thought about that. They've never thought: actually, you've got a cycle, you're different. So if I do succeed at all, that's what it's down to."

In the winter of 2010, a few months after his wife's diagnosis, Self noticed his hands becoming swollen and livid. It transpired he was suffering from polycythemia vera, a potentially fatal blood disorder in which the bone marrow produces too many red blood cells. The treatment requires the regular siphoning off of blood from the veins through a process called venesection.

"It's not bad actually," he says. "I had a pint out yesterday. Last year I was having two pints a week taken out and… it felt very traumatic. But they've sort of stabilised it and it's now only every four to six weeks."

In an essay he wrote for Granta last October, Self explained that part of the trauma of the treatment stemmed from his well-documented experiences of drug addiction. "I first stuck a needle in my arm in the summer of 1979," he wrote. "I was 17 years old. I often think back with a protective tenderness towards my younger self and wish I were somehow able to dissuade him from such a mutilation, from breaking the blood-air barrier in that crazy way."

Why did he start injecting?

"I arrived at it through being very, very unhappy," he says now. "You can never run your life through again and say, 'Right, now let's put you in a different environment and see whether you're still [unhappy].' Do you know what I mean? There is no counter-life in that way. I suspect it's a bit of nature, a bit of nurture, like most things."

Self was raised in Hampstead Garden Suburb. His mother was a Jewish American who worked as a publisher's assistant, his father a professor of public administration at the London School of Economics. His parents divorced when Self was 18 and his father later emigrated to Australia.

When Self started doing hard drugs his mother sent him to a psychiatrist – giving rise to one of the recurring obsessions of his novels. Zack Busner, the psychoanalyst in Umbrella, is a familiar figure, having appeared in several of Self's previous works as a self-promoter intent on making his name at the cost of true engagement with his patients.

"My mother loved psychiatry," says Self. "You've got to remember that, in the immediate postwar period when my mum was a young woman, Freud hit America like a film star. The whole nation went belly-up to it. For her, attempting to treat mental distress in some way, shape or form was just the done thing – [it] absolutely proved you were a cultured person."

Self, by contrast, was deeply influenced by the work of RD Laing: "That idea, really embodied in anti-psychiatry, that madness was just a different way of being, man, and let's all just hang out. And that just merged in my own head with getting off my face on drugs."

For years he injected heroin and also took cocaine and amphetamines. He went to Oxford to read PPE, graduating with a third after spending much of his spare time "hanging out" with schizophrenic outpatients from a local hospital. There was a brief period of cold turkey in the 1980s but he continued to use until a spectacular fall from grace in 1997 when he was found snorting heroin on John Major's jet while covering the election campaign for this newspaper.

Is his creative impulse allied to an innate self-destructiveness?

"Not any more. When I stopped drinking and doing drugs I let go of that… If I'm frank, one of the reasons I stopped eventually was because I could feel I would be unable to write. Which sounds awful because, of course, I should say I stopped for my family but it's a very, very all-consuming thing, the old writing."

Now that he has four children of his own – his two eldest, Alexis, 22, and Madeleine, 19, are from his first marriage to Kate Chancellor – does he worry they could have inherited his capacity for unhappiness?

"Put it this way: none of them seems to be as radically unhappy as I was. And for that, I'm grateful."

Although Self had an ambivalent relationship with his parents, much of the inspiration for Umbrella came from his own family history. In the book, Audrey's older brother, Albert, a brilliant mathematician who ran the Woolwich Arsenal, and her younger brother, Stanley, who dies in the trenches, are directly based on Self's paternal grandfather and great-uncle.

"My grandfather was a polymath and savant," says Self. " My grandparents lived in Brighton, and if you came back from the front he'd say, 'I calculate your pace to be 26 inches, therefore you took 2,923 paces.'

"In a kind of tedious middle-aged way I was doing a bit of family history and saw that Albert had a younger brother, Stanley, which was never spoken of when I was a child. He's in the 1911 census, and I thought, 'Hmm, he must have died in the first world war.'"

There seems to be a current vogue for first world war novels and TV adaptations. Did he watch Downton Abbey?

"I certainly watched a bit of the first series and could enjoy it just for the stage-dressing… But by the time it got to the second series it was so kind of leaden and formulaic that it was impossible to watch."

I'm not particularly astonished to hear Self didn't take to the light campery of prime-time period television. He is not a man who likes to make things easy for himself. At one point, eager to explain how he tackled the extensive research for Umbrella, he leaps up to show me a black notebook. It is filled with yet more scribbled Post-It notes organised into sub-categories under separate headings for "Metaphors", "Tropes" and "Ideas". He explains that the entire novel is shaped like an umbrella – with curved spokes of narrative radiating outwards from a central scene.

He is already beavering away at his next novel, the working rubric for which is, he tells me, "Jaws without the shark". He rolls a cigarette and lights it, taking a drag and blowing out smoke with a wry smile, aware of how absurd this might sound. For someone with such an inquisitive drive, he is surprisingly easy to be around – solicitous, funny and kind. He would say, of course, that the world cannot be understood through simple observation. He might be right: I'm not sure anything could fully explain what goes on in Will Self's brain.

Still, he makes a great cup of coffee.

Umbrella is published by Bloomsbury on 30 August.