As the Guardian’s fiction editor, I’ve been writing about the Booker prize for years. But this time, I was privileged to go behind the curtain, as part of the judging team along with artist and writer Edmund de Waal, musician Nitin Sawhney, and writers Yiyun Li and Sara Collins. The longlisting stage was a delight: choosing the books we wanted to spotlight. (Though of course we’d happily have produced a directors’ cut of 30 books or more.) Losing some of them at the shortlist meeting was harder, though as we talked about each book, unanimity fairly quickly evolved. Each of these six novels does something unique; weighing them against each other to find a winner will be trickier still.

James by Percival Everett feels like the book he was born to write. It’s a reworking of Mark Twain’s Adventures of Huckleberry Finn, a toxic foundation stone of the American literary canon, but it would not be overpraise to say it supplants it. The novel is told from the perspective of the enslaved Jim, who in the original is both a lesser creature and a learning opportunity for Huck to demonstrate his own humanity. Everett fills in the gaps, to show the hell behind Twain’s easy humour. His James saves the folksy slave dialect for whites, and talks very differently among his own people. Throughout his career Everett has explored the construction of race through cliches of thought and language, and that fascination reaches its full expression here. He’s always played with genre, too: James combines satire, parody, adventure story, children’s fiction, horror. And he makes it look easy – it’s ferocious, but ferociously readable too.

Samantha Harvey’s Orbital is a slim but beautifully rich novel. You could read it over and over, just as watching a sunrise never gets dull – something the characters in her novel, six astronauts orbiting 250 miles above the Earth’s surface in the International Space Station, know only too well. In the 24 hours of the book, they see 16 sunrises and sunsets. Orbital, which Harvey has described as “space pastoral”, is tightly constructed yet also breaks free of time: its prose is both scientifically precise and transcendent. Harvey gradually conveys what it means for the astronauts to exist in a constantly moving closed system where everything must be shared and reused – and leaves it to the reader to infer that the same is true of our home, the fragile, irreplaceable Earth. You could call it cli-fi, but Harvey’s urgent ecological message is conveyed through love letter rather than dystopia.

Creation Lake by Rachel Kushner is a different sort of joyride. A philosophical novel of ideas wrapped up in a spy caper, it’s the story of an American provocateur who arrives in rural France to disrupt a commune of eco-activists – and despite her ultra-cool confidence, things don’t go to plan. The prose is thrilling, and the audacious character of “Sadie Smith” – a kind of steely self-willed superwoman, whose veneer begins to crack as the novel goes on – irresistible. The novel’s themes are certainly current: radical resistance to the corporate exploitation of natural resources, the manipulations of power and politics, the emptiness of a post-truth age. But Kushner’s questions are also timeless and profound: where do we come from? What is an individual really made of? How do we construct reality? She brings it all together with amazing flair – and fun.

Held by Anne Michaels is an extraordinary reading experience – it seems to transport you to another state of consciousness. Spare and elliptical, it alights on many intense stories, occurring over several generations, but the years and distances between them are just as important. It’s chiefly about human connection, and that old chestnut, the power of love. The novel unfolds against a 20th century of war and trauma in which scientific advances open up new spaces of doubt and revelation. As the book repeatedly demonstrates, nothing is permanent, yet in the context of a human life, some memories are indelible. Michaels touches profundity through fleeting moments: as one character says: “Limit is proof of the beyond.”

In Charlotte Wood’s Stone Yard Devotional, an Australian conservationist withdraws from the world to a monastery. She’s not religious, but something about the steady acts of service and observation that make up the nuns’ days, months and years anchors her there. Her friends are furious that she has given in to despair in the face of the climate crisis – given up on any attempt to make things better. There are vast, painful undercurrents in this apparently quiet book, difficult questions about ethics and faith and grief and personal responsibility. And, of course, there’s no escaping the world: global heating creates a plague of mice that terrorises the monastery, and nuns are people too, as awkward and complicated and baggage-laden as anyone outside. Wood brings a wry, humorous, remarkably light touch to her meditation on what humans can bear, and how we do or don’t survive.

The Safekeep by Yael Van Der Wouden is a brave and thrilling debut about facing up to the truth of history, and to one’s own desires. It’s set in the Netherlands, 15 years after the end of the second world war, amid a traumatised society in denial about its failure to reckon with the fate of Dutch Jews. Prickly, buttoned-up Isabel – a fantastically sour and spiky character – lives alone in the family house after the death of her mother, clinging to routine and disapproving of her brothers. The first is gay, the second an unreliable romantic who sends his girlfriend to stay. The tightly controlled narrative breaks open into a tale of erotic awakening, followed by an almighty twist. Anticipating that twist will not lessen its impact – fittingly enough for a novel in which everyone knows more than they can bear to admit. Van Der Wouden brings stunning power and control to her page-turner about trauma and repression.

• The 2024 Booker prize winner is announced on 12 November. Explore the shortlist and, for a limited time, get all six books for only £75 at guardianbookshop.com. Delivery charges may apply.