This might seem like an eccentric book. As Labour prepares for power after four consecutive general election defeats, Andy Beckett is interested not in what is to come but what has just been. He is particularly preoccupied by the rise of Jeremy Corbyn, what happened to him as party leader and what his leadership represented. The Searchers is mostly fair-minded, diligently reported and researched, but leaves you in no doubt that Beckett, a Guardian columnist, is a sympathetic Corbynite.

In the long, final section, covering 2015 to the present day, Beckett writes nostalgically about the excitement of the early years of Corbyn’s leadership when the left, for so long ridiculed, traduced and marginalised (Peter Mandelson joked during the era of New Labour dominance that they had been contained in a “sealed tomb”), seized control of the party and unlocked a spirit of radical countercultural optimism, especially among younger voters.

The author misses those days but in seeking to recreate the social atmosphere of the country during that period he is perhaps too forgiving of Corbyn’s failures, anti-Zionist zeal and toxic associations. Not forgetting his crass stupidity. This was demonstrated, most emphatically, by his response to the Novichok nerve agent poisoning of Sergei Skripal, a former Soviet military intelligence officer, and his daughter in Salisbury in 2018 when the Labour leader gave the benefit of the doubt to Putin’s Kremlin.

In 2023 Beckett attended a Bernie Sanders event at the Royal Festival Hall in London. The cross-class social mix and febrile mood reminded him of old Corbyn rallies. “The audience was multiracial, male and female in roughly equal proportions … It did not look like a Britain whose political time had gone for good.”

But London is not Britain. The Corbyn events that Beckett attended while reporting for the book were invariably in London – or Bristol and Brighton, cities with similar demographics to the vibrant multicultural capital. We never encounter him in those faraway Brexit-supporting former Labour towns in which Corbyn and his movement were loathed.

But this is not just a book about Corbyn and Corbynism. The Searchers has larger ambitions and more broadly is an absorbing history of Labour’s radical left from the late 1960s to its present marginalisation. It is also a series of interconnected mini-biographies – of Tony Benn and the four prominent politicians who were inspired by him: Ken Livingstone, Diane Abbott, John McDonnell and Corbyn.

Beckett’s Gang of Four (as I shall call them) were all products of the London left. Livingstone and McDonnell worked together in the 1980s, respectively, as leader and deputy leader of the Greater London Council (GLC), which championed municipal socialism and delighted in tormenting Margaret Thatcher. Her response: she abolished the GLC in 1986.

Abbott, a former TV journalist, reported on the GLC and later worked as a press officer for it. She was encouraged by its embrace of minority rights and identity politics. The GLC back then was widely mocked as “loony left”, but its equity, inclusion and diversity policies are now mainstream.

Corbyn, who later had a relationship with Abbott, moved to London after working with the Voluntary Service Overseas scheme in Jamaica and then travelling through Chile, where he observed the rise of Salvador Allende, the elected socialist president whose Popular Unity government was toppled in a US-backed coup in 1973. (The fall of Allende was a cautionary tale for the internationalist left.)

On his return to England, Corbyn, who had messed up his A-levels, studied desultorily at North London Polytechnic before working for several trade unions. He became the MP for Islington North in 1983, the same year that Tony Blair and Gordon Brown were elected to parliament (Livingstone and Abbott were elected in 1987; McDonnell in 1997).

The Gang of Four shared a politics and were inspired by Benn’s campaign to empower members and activists in pursuit of “change from below” and greater party democracy. They supported Benn’s attempt to win the deputy leadership in 1981, a bitter sectarian struggle against Denis Healey that poisoned Labour for a generation. As a restless, technocratic senior minister in Harold Wilson’s governments, Benn had become disillusioned with the resistance his reforming ideas had encountered, from within the civil service and his own party. Through the 1970s he moved further to the left.

The Gang of Four were “Bennites” but had different styles and methods. Livingstone was the Machiavel of County Hall, a pragmatist and dealmaker. “There’s no permanent friendships [in politics],” he told Beckett. He was ruthless in achieving what he wanted but, in the end, was a local rather than national hero. His rise in the parliamentary party was thwarted by Neil Kinnock. “He hated the London left,” Abbott said of the former party leader.

McDonnell was an uncompromising ideologue and, according to Beckett, a Gramscian. He was an autodidact, having gone to Brunel University as a mature student. He was from an Irish-Catholic Liverpudlian family and considered becoming a priest. He was patient and relentless and would later emerge as the de facto leader of the parliamentary left and chair of the Socialist Campaign Group, which is now a demoralised faction.

McDonnell was the Corbynite who impressed me most when I interviewed him. He was interested in big ideas and political economy. He was a hard worker, shrewd strategist and desperately wanted to win. And he had a theory of history and of how he wanted to transform the state. For McDonnell, Corbynism was a counter-hegemonic project. For Corbyn himself, the accidental leader, not so much.

The young Corbyn was a rabble-rouser and agitator. He was intoxicated by the upheavals and student rebellions of the late 1960s and the world of leftwing politics as he found it in London: the radical magazines, the rallies, the festivals, the demonstrations. He relished protest but was a reluctant frontman and had intellectual insecurities. He agreed only to run for the leadership in 2015 because both McDonnell (twice) and Abbott had tried and failed before him, and the Socialist Campaign Group wanted to be represented in the contest. During a brilliant campaign, he unequivocally rejected austerity and spoke directly in a manner that inspired activists who were weary of the tortured triangulations of senior Labour politicians. Corbyn won convincingly but was overwhelmingly rejected by the parliamentary party. A long civil war had begun.

Abbott was a true pioneer, a state school Cambridge graduate and the first black female MP. She was an adept media performer as well as being a passionate campaigner for racial, sexual and gender equality. Perhaps no other modern MP has suffered more sustained abuse and vilification and today Abbott is an unhappy exile within the party. She, like Corbyn, is stained by accusations of antisemitism.

The book shares a title with John Ford’s great western, from 1956, starring John Wayne as a Confederate veteran of the American civil war. He embarks on a quest, in the desert landscape of west Texas, to find his niece whom he believes has been kidnapped by members of the Comanche Native American tribe. What were the searchers of Andy Beckett’s book looking for?

They were looking for a lot of things but most strikingly for an enduring socialist alternative. They were opposed to rampant capitalism, entrenched privilege, social conservatism and American hegemony. They were economically statist and socially ultra-liberal. On social and cultural matters, Beckett writes, “the left has won so many battles that its victory has become invisible”.

So why the ultimate disappointment, why the sense of lost opportunity that flows like an underground stream through the book? The answer, I think, is that when it mattered most, when they had the opportunity to lead the Labour party to power and effect the political and economic transformation for which they had long campaigned, Corbyn, McDonnell and Abbott were abjectly defeated by Boris Johnson, the clown prince of Tory politics. They told the people who they were and what they wanted and the message in return was: “Enough. No more!” The tomb had been resealed. It is now left for Keir Starmer and Rachel Reeves to lead Labour into a new era of government.

Jason Cowley is editor of the New Statesman

• The Searchers: Five Rebels, Their Dream of a Different Britain, and Their Many Enemies by Andy Beckett is published by Allen Lane (£30). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply