What do we talk about now we no longer talk about books? Well, everything else, of course. Since George Saunders won this year’s Man Booker prize last Tuesday evening for his novel Lincoln in the Bardo, virtually every report and a heap of internet chatter besides has led with the fact of Saunders’s Americanness. This, put together with the Americanness of last year’s winner, Paul Beatty, and the Jamaicanness of the previous year’s, Marlon James, seems to spell gloom for the UK.

It’s as though Colin Welland’s Oscar acceptance speech – “The British are coming!” – is being rewound at zanily high speed. That we did this to ourselves in 2014 – allowing the Yanks in to take their chances alongside the Brits, the Irish, the Indians, the New Zealanders and more – hovers in the background, as if in reproof to openness. Among the objections to the widening of the rules is the plaintive cry that British writers should be let loose on American literary prizes, which has a certain amount of merit, but is also somewhat beside the point: you might be nice enough to share your toys, but there’s no onus on the friends you invited to tea to share theirs.

Hastily leaving my seat at the medieval Guildhall on Tuesday to bolt into a waiting radio car, I found myself taking part in a curious – and somewhat spiky – exchange. Had the Booker become, wondered the Radio 3 presenter, who was admittedly attempting to slot it into to a programme about national identity – a neocolonial enterprise? Was it, indeed, just part of a scene, like the Henley Regatta?

If this is an agenda that seeks to root out privilege and elitism, the Booker presents it with a problem. Here are the winners from the last 10 years: Anne Enright, Aravind Adiga, Hilary Mantel, Howard Jacobson, Hilary Mantel again, Eleanor Catton, Richard Flanagan, James, Beatty and Saunders; a spread of nationalities, ethnicities and class backgrounds, with, as far as I am aware, not a single Etonian in sight. And what’s particularly striking in the case of the last three winners is how long they wrote in relative obscurity before receiving anything approaching widespread recognition, praise and, of course, financial security.

But the quarrel, of course, is not with the individuals, it is with the system and prizes, especially rich, corporate-sponsored, high-profile, swishy ceremonied ones, are a convenient focus for our cultural dissatisfactions, fury and angst.

And there is much to criticise. The Booker, for example, requires the publishers of longlisted authors to make commitments about print runs and stock levels that are a walk in the park for larger publishers, but hugely daunting for the small, independent presses that are playing such a vital role in encouraging talent. If one of those publishers should find themselves with a shortlisted title, they must find £5,000 to help with publicity and hope that they will see a return on their investment.

I recently spent an evening at a small literary festival, at which the organisers, not, probably, awash with excess cash themselves, took a group of us out for a jolly pizza and pasta supper at the end of the day’s events. I found myself chatting to a recent Booker shortlistee whose work is published by an aforementioned microscopic publisher. He regaled me with tales of their trip to London to take part in Booker business and the comic cognitive dissonance between getting on your black-tie gladrags, carousing with literary high society and then returning to the backpackers’ hostel afforded by your budget. There was not a shiver of complaint in his account, by the way, just pure delight.

But back to the radio car. Did I, asked the presenter, think Saunders’s Lincoln in the Bardo was a worthy winner? I did, I replied, and if only I had left it there. In fact, I thought it was wonderful. Wonderful. I may have added a third wonderful. In my defence, I had emerged from a pressure cooker of hyperbole. Adjectives were, to tax the metaphor a moment longer, merely steam escaping from the bubbling casserole of literature.

He wasn’t having it. Surely, I simply meant it was good. And if I were to call Lincoln in the Bardo wonderful, what would that leave for War and Peace?

Readers, this had me stumped: nobody expects the Russian inquisition and, really, I had nothing smart to say about Tolstoy. But I managed a response and we moved on.

Luckily, I have now had several days in which to channel my esprit de l’escalier. And in fact, it seems remarkably simple to me. I do think Lincoln in the Bardo is a wonderful book. I thought so when I reviewed it and called it “a brilliant, exhausting, emotionally involving attempt to get up again, to fight for empathy, kindness and self-sacrifice, and to resist”. I thought so even more when I read it a second time and more again when I listened to the audio version that brings to life its 166 voices. I think so when I contemplate the strangeness of its story – a dead child, who happens to be the son of Abraham Lincoln, wandering through a cemetery, waiting to be gathered to the afterlife – and the utterly uncompromising nature of its execution.

When I very first picked it up, I thought I would never get to grips with its hypermobility, its shifts of voice and aspect, its ceaseless wrong-footing. And alongside all this, its depiction not only of a father in acute grief, but of a man leading a nation in the crisis of civil war, its fields littered with countless other dead sons. It’s impossible to read Lincoln in the Bardo and not think of America’s current convulsions, of the impossibility of reconciling personal and public duty, of the harrowing, hollowing nature of irreversible loss. So what I might have replied to the Tolstoy question is: “But it is War and Peace.”

None of this is to say that you have to like it. It might leave you cold. It might irritate you. You might be reading something else that you’re enjoying more. If you are, that is excellent news. Books, and the prizes that reward them, should not be tests we pass or fail. The publishing and literary worlds have, in recent years, begun to turn the tanker around and address issues of representation, the stories untold and voices unheard. Changes will not happen overnight, but there is one change every reader can make instantly: we can start to talk about what’s inside the books, not everything that’s outside them.