What do book prizes have to do with serious literature? On the long view, the answer is: not much. Even the immensely distinguished Nobel prize, which has just delighted many British readers with its choice of Kazuo Ishiguro, has logged some pretty forgettable selections. Who, for instance, still reads Rudolf Christoph Eucken (1908), Karl Adolph Gjellerup (1917), Frans Eemil Sillanpää (1939) or Nelly Sachs (1966)?

In the arts, prizes will always be a lottery. Posh panels are just as vulnerable as the rest of us to the vagaries of taste. The joy of reading is that it’s free from the thought police. Books live and die in the hearts and minds of readers. Still, for close on 50 years, Man Booker’s quixotic efforts have made surprisingly good sense of a difficult, even impossible enterprise.

Taste and judgment aside, the other intractable dimension of the Booker conundrum is its annual rendezvous with the marketplace. As in farming, or finance, there are good years and bad years. 2015 and 2016 were good years with excellent winners (Marlon James’s A Brief History of Seven Killings; Paul Beatty’s The Sellout).

This year, Booker’s panel is up against it. Before we discuss the shortlist, we have to note that, for whatever good reason, the judges decided to exclude some powerful contenders: Sebastian Barry (Days Without End), Arundhati Roy (The Ministry of Utmost Happiness), Zadie Smith (Swing Time) and, perhaps most surprising of all, Colson Whitehead (The Underground Railroad).

When Lola Young, as chair, summarised the shortlist as “unique and intrepid books that collectively push against the borders of convention”, she articulated a mission statement for a final session that promises to be an excruciating visit to the third circle of a literary critical inferno.

In practical terms, Young and her team have to choose between three Americans, an English woman, an award-winning Scot and a British-Pakistani. Collectively, their novels have been described by one of the judges as “transcultural”, a word still new to many dictionaries.

First on the shortlist is Paul Auster’s 4321 (Faber), a consciously epic, quasi-cubist portrait of an American boy’s life during the second half of the last century. (Full disclosure: I was Auster’s first publisher and have been an admirer of his work ever since The New York Trilogy.)

4321 offers a hefty punch from an American heavyweight. At more than 1,000 pages, the multi-form versions of the life of Archibald Isaac Ferguson demonstrates a major writer at his best – original, ambitious, pitch-perfect and wholly absorbing. This is a novel tormented by fate, contingency and playful literary experimentation. But it’s also a narrative to get lost in, a Bildungsroman that is, by chance, a poignant elegy to the America that is being trashed by Trump and his vandals. 4321 must be a real contender.

Inevitably, Auster’s book towers over Emily Fridlund’s History of Wolves (Weidenfeld), an enigmatic coming-of-age novel set on the edge of a lake in the American midwest. Some readers may compare Fridlund’s book unfavourably with Marilynne Robinson’s Housekeeping, which is also set by a lake, but History of Wolves is grounded in a more sinister domestic narrative that grips the reader with a fierce vision of adolescent estrangement. It’s a powerful debut.

From these all-American narratives, we move to Britain with Fiona Mozley’s Elmet (John Murray), another first novel about secrets in the family. As her title suggests, Mozley has drunk deep at the fount of Ted Hughes and has written a dark hymn to the landscape and language of Yorkshire that’s also, like Hughes’s poetry, bleak, lyrical and steeped in blood.

Where Mozley draws on an English tradition, in Exit West (Hamish Hamilton), Mohsin Hamid strikes a global dystopian note that’s fresh, contemporary and audacious, a love story set in a disintegrating society; a novel for the moment.

Which brings us to the big American challenger here, Lincoln in the Bardo by George Saunders (Bloomsbury), the bookies’ favourite. Saunders has been extravagantly praised in the US for his short fiction, notably CivilWarLand in Bad Decline (1996), and is, unquestionably, a highly original talent who is taking fiction into extraordinary new territory. Lincoln in the Bardo, his first novel, is a strange and haunting reverie on Abraham Lincoln’s grief at the death of his 11-year-old son, Willie, “a small mirror of himself”, in 1862. The “bardo” is the Tibetan Buddhist limbo between death and rebirth in which Lincoln’s dead son meets ghostly voices from the past in a dazzling collage of voices.

You could describe this as a postmodern historical novel, but that would be to underestimate the thrill of Saunders’s intentions. Braiding snatches of prose, both original and archival, he redefines his chosen genre with some outrageous comic touches and many arresting moments of sheer magic. Experimental yet firmly anchored to its readers’ attention, it offers the kind of satisfactions typical of many previous winners. If it had not been so wildly acclaimed, it would seem a shoo-in for this year’s prize. And yet…

There is, from the final book on this list, Ali Smith’s Autumn (Penguin), some serious competition. Smith’s is another highly acclaimed performance, the first of a four-part series about time and the seasons. Autumn is a bittersweet tour de force set in “the worst of times” – Brexit Britain – and interweaving art, death and the mysteries of love. Smith’s portrait of centenarian Daniel Gluck, approaching death, and his passionate protege, Elisabeth, is a stunning exploration of memory and regret, a book that celebrates and entertains as much as it takes us into the twilight world of the dying. On some readings, Autumn has everything one would want from a winner: humanity, brilliance and a kind of subversive ecstasy.

Who knows how the judges will adjudicate this list? Almost any outcome is possible on 17 October. Who knows if what they choose is literature? That’s for posterity. My hunch is that a US writer will take the prize again, no doubt stirring up more anti-Americanism, though Smith will have a lot of home support. You never can tell.

• To order all six shortlisted books for £50 (RRP £76.94) go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99

• This article was amended on 16 October. Mohsin Hamid is British-Pakistani, not Pakistani as originally stated