“Madame Bovary, c’est moi,” Flaubert said. What would Will Self say of his new novel? “Elaine, c’est moi”? “Madame Bovary, c’est ma mere”? Described in the blurb as “perhaps the first work of auto-oedipal fiction”, Elaine is based, we’re told, on the diaries of Self’s mother, Elaine, so it promises just the sort of ping-pong game of identification and countertransference a writer so steeped in psychoanalysis is likely to revel in. It also puts a writer whose usual beat has been the grimy pubs and crumbling mental hospitals of London into the territory of Richard Yates or John Cheever.

It’s the mid-1950s, upstate New York. Elaine is a thwarted writer, unhappily married to John Hancock, an ambitious Milton scholar jockeying for tenure at Cornell. Elaine once studied poetry under Theodore Roethke, but now she’s a home-maker, a faculty wife, trapped by potlucks and pot roasts and the pressing of shirts (she calls the ironing the irony). She suffers migraines, severe PMS and hysterical attacks. She drinks too much. She has been quietly sacked by her therapist, Dr Freudenberg (really), and she chews over that rejection while angrily rejecting his psychoanalytic nostrums. She has imagined killing their toddler, Billy, before killing herself. And she lusts, semi-hopelessly, humiliatingly, after other men: specifically, when we meet her, John’s swarthy and assured colleague Ted Troppmann.

The Hancocks’ social circle in Ithaca consists of bluff, competitive male academics and their brittle, gossipy wives. There’s an unexpected walk-on from Vladimir and Véra Nabokov (the former with “neat, aquiline nose and hooded eye”; the latter in a frayed squirrel coat), whose conversation with Elaine is interrupted when John drags her off to suck up to a departmental bigwig. Affairs, or drunken necking at parties, are the norm. Everyone’s “tight”: “There are the sloppy grins of fast, fake friends, and a sort of haven in the hubbub – albeit one palisaded by their own hilarity. Rose cackles he-he-he-he and the man back-cackles he-he-he-he, and the woman joins in a-ha-ha-ha-ha-ha, which means John has to h’h’h’h’h’.”

The allusive possibilities of the setup are allowed to make themselves felt. Cornell really is in Ithaca (coincidentally, per Tennyson’s Ulysses, a place of domestic dissatisfaction: “by this still hearth, among these barren crags”), and the poisonous family home is on Hemlock Street. Karma, here, isn’t a bitch, but the name of the family cat. And that the paterfamilias is a Miltonist – whose manuscripts Elaine types out for him, in an echo of Milton’s amanuenses – supplies another chord of resonance (Will Self’s real dad was a town-planning scholar).

And then there’s little Billy: precocious, “unnaturally attached to his father”, he is to be found even as he approaches puberty hoarding the clipped-out funnies from the newspapers, slurping brownie-batter, or dressing up reluctantly for a holiday in a home-made pilgrim costume “rather than his preferred outfit of felt Stetson, plastic holsters, and a pair of plastic six-shooters loaded with caps”. Can we detect a short-trousered version of today’s lugubrious novelist? Occasionally, though, “his sunny features become overcast with what she suspects is his fundamental uncertainty about the world and his place in it”.

Here, then, is an intense and claustrophobic portrait of an intense and claustrophobic world. Self’s later-career stylistic mannerisms are in full effect: the ellipses, the grotesque similes, the end-of-sentence bursts of italics. The timeline is jagged, and as Elaine’s mental state degenerates, dreams, hallucinations, fantasies, episodes of body horror and the smeared perception of the intoxicated jumble in the narrative. Self has many qualities, but a light touch has never been one of them.

There aren’t many laughs, either. It’s a slog. And those looking for direct, real-life gossip should be warned that it diverges substantially from the historical record. But some of the set-pieces – a shocking burst of domestic violence as the Hancock marriage hits a crisis point, for instance – have real force. And in the intensity with which Self inhabits an Other, so to speak, and with which he brings to life Elaine’s tormented and disordered mind amid a grimly patriarchal society beset by wider societal neuroses and hypocrisies (the colour bar; the bomb; the red scare; her shame at her Jewish origins and John’s anglicising of his German name), there is much to admire.

• Elaine by Will Self is published by Grove Press (£18.99). To support the Guardian and Observer order a copy at guardianbookshop.com. Delivery charges may apply