Robert Coover, who has died aged 92, was one of the pioneers of postmodern writing as it developed in the US in the 1960s, alongside writers such as Thomas Pynchon, John Barth, John Hawkes and Donald Barthelme. This was before postmodernism acquired its current political and structural connotations, a time Hari Kunzru described, in a 2011 Guardian profile of Coover, as a “fantastical funhouse of narrative possibilities”.

Coover recounted his frustration with Saul Bellow’s 1953 novel The Adventures of Augie March, and what was called realism. “It wasn’t realistic. He was using modes of response to the world that had become stultified.” He later explained: “I learned my realism from guys like Franz Kafka.”

Despite success with his early novels, his shorter work was more defining, in particular his 1969 collection Pricksongs and Descants (the title refers to how, in manuscripts of medieval European music, the notes were physically “pricked” or marked with dots). It became a crucial work of postmodern fiction, particularly The Babysitter, whose free-standing paragraphs challenged the nature of story-telling.

It began with a teenage girl’s minding of three children and escalated into a series of lascivious, sometimes contradictory, vignettes involving violent boyfriends and parental fantasies. The seemingly random order of the narrative was reminiscent of Julio Cortázar’s Hopscotch or William Burroughs’s Naked Lunch. As Barth wrote: “The process is the content, more or less.”

Improbably, The Babysitter became a 1995 Hollywood film starring Alicia Silverstone; the result, closer to a Jamie Lee Curtis horror film emphasising the salacious, was itself worthy of another Coover fiction.

HIs biggest, and most controversial, popular success was his third novel, The Public Burning (1977), a reimagining of the trial and execution of Julius and Ethel Rosenberg for passing atomic secrets to the Soviet Union.

A blovatious, cartoon-like Uncle Sam battles the villainous Phantom, in chapters alternating with Richard Nixon’s schemes to stage the execution in Times Square. It reached the bestseller lists, but the publisher worried about libel suits, and let it go out of print.

This was a long way from Coover’s first novel, The Origins of the Brunists (1966), a sprawling tale about a cult built around the sole survivor of a mining disaster. It won the Pen/Faulkner award, and harked back, like Pynchon’s V (which also won the Faulkner) or Barth’s Sot Weed Factor, to the playful narratives of 18th-century novelists such as Henry Fielding or Laurence Sterne.

But it did not prepare readers for his second novel, The Universal Baseball Association, Inc, J. Henry Waugh, Prop. (1968) in which the titular lonely accountant creates a baseball simulation game, rolling three 10-sided dice and using the 1/1,000 probabilities to recapitulate imaginary players’ action in games. When three rolls demand a shocking event involving his favourite created player, his world collapses. Tightly written and set in a controlled, though random, environment, the novel became a cult classic.

Coover’s background was midwestern traditional. Born in Charles City, Iowa, he was the son of Maxine (nee Sweet), a doctor’s assistant, and Grant, a newspaper editor. The family moved frequently, and Robert edited school papers both in high school and college.

He transferred from Southern Illinois University to Indiana University, graduating in 1953 with a BA in Slavic studies. He then joined the US Navy as a lieutenant, serving mostly in Europe; there were later suggestions he was in intelligence.

Discharged in 1957, he spent time writing in a cabin in Canada. In 1959 he married Maria del Pilar Sans, a Catalan needlepoint artist whom he had met in Spain.

He received an MA in humanities from the University of Chicago in 1965, and began teaching at Bard College; he would teach at five other universities and spend three years as fiction editor of the Iowa Review before Hawkes brought him to Brown University, in Rhode Island, in 1981, where he stayed until 2012. His students included many successful writers, among them Jeffrey Eugenides, Jim Shepard, Sam Lipsyte and Joanna Scott.

In 1968, Coover joined in the protests against the Vietnam war. With Nixon running for president, he also wrote a novella, The Cat in the Hat for President, which his publisher rejected because Dr Seuss was their bestselling author.

But Coover’s editor passed the manuscript to the New American Review, who published it; it eventually appeared as a book in 1980 retitled A Political Fable. He revisited Nixon one more time, in Whatever Happened to Gloomy Gus of the Chicago Bears (1987), where Nixon’s devotion to football as a college bench-warmer is transformed into a tale of obsession. Coover claimed Pynchon’s V had convinced him of the importance of humour; “all my work is basically comic,” he said.

Coover was also a football fan, of FC Barcelona (his wife’s influence) and Queen’s Park Rangers, a loyalty picked up in the 70s while in Britain on a fellowship. There he bonded with Angela Carter, as can be seen in his reworking of fairy tales in the novellas Briar Rose (1996) and Pinocchio in Venice (1991). In his introduction to Briar Rose, John Banville wrote: “Coover is a magician and these are his spells.”

Briar Rose was published alongside Coover’s novella aping Victorian sadomasochism, Spanking the Maid (1982). He found genre fiction, including pornography, “though conservative in form”, more useful for a writer trying to “burrow inside the collective psyche”. Gerald’s Party (1996), an outgrowth of The Babysitter, spiced up the classic mystery, while Noir (2010) features a detective called Phil M Noir.

Coover was fascinated by the paradox of writing being a two-dimensional exercise in a three-dimensional world. At Brown, he helped found the Electronic Literature Organisation to explore the digital medium. His works were often described as “metafictions”, but he began teaching students the art of “cave-writing”, the cave being an immersive 3D space in which they could produce “hyperfictions”.

“What I saw quite clearly in the 80s, before the internet, was that the whole world was shifting to digital formats,” he wrote in a 1992 essay The End of Books.

Nevertheless, he continued writing: his later work included a 1,000-page sequel to his first novel, The Brunist Day of Wrath (2014) and Huck Out West (2017), a follow-up to Huckleberry Finn.

He admired graphic novels such as Alan Moore’s Watchmen, and his 2021 story Street Cop was illustrated by Art Spiegleman. He leaves more than 30 uncollected short stories, many of which were published in New Yorker magazine.

When asked who he would like to write his life story, Coover replied: “Samuel Beckett. It would be funny and mercifully short.”

Maria survives him, as do their daughters, Sara and Diana, and a son Roderick.

• Robert Lowell Coover, writer, born 4 February 1932; died 5 October 2024