In cities such as Kabul, Herat and Bamyan, you feel a profound sense of shame when you see the after-effects of the west’s abandonment of Afghanistan: the long queues for food; the people persecuted and sometimes killed for trying to defend the freedoms they have gained in the past 20 years. Only six months after the fall of Kabul, the American and British media seem mostly to have forgotten about it. Typically, the judgment seems to be that we should never have gone in to help Afghanistan in the first place: as though that solves anything.

The catastrophe began before the 2020 US election, when Donald Trump was casting around for anything he could claim as a foreign policy victory, following a series of failures with North Korea, Russia and China. Afghanistan meant nothing to Trump, and he handed it to the Taliban on a plate. There was no obligation on Joe Biden, when he became president, to follow through with this embarrassing deal; but he was desperate to show that he too put American interests before anyone else’s, and he allowed it to stand. After all, when does a US president put a small country’s needs before the chance of an uptick in the domestic opinion polls?

The disaster in Afghanistan shows up American and western weaknesses in a way that not even the defeat in Vietnam did. A powerful new collection of essays, Losing Afghanistan, points out again and again that people around the world now see the US as a fickle ally. What, for instance, does it say to China at a time when Xi Jinping is mulling over the invasion of Taiwan? One of the best pieces here comes from a former British diplomat, Nick Fishwick, who served in Afghanistan: he blames the west’s failure on weak leadership, an ignorance of the country, short tours of duty and a desire to compete with one’s predecessors. He’s right. Graham Cundy, who served with the Royal Marines there, speaks with insight about British mistakes in Helmand Province.

But this wasn’t so much a failure of the men and women on the ground as of politicians back in Washington. There are valuable contributions here on the reactions of Afghanistan’s near neighbours, Pakistan and India, and on the Russian experience. (Russia comes out of this study reasonably well; it has maintained a steady interest in Afghan affairs ever since its withdrawal from the country in 1989, yet has been careful not to interfere in its politics.) There’s less detail on China’s approach. In Kabul recently, I spoke to the deputy finance minister, Nazir Kabiri, who was a member of the last regime and stayed on to work with the Taliban out of sheer patriotism; they have scarcely anyone who understands economics. Kabiri told me the Taliban would like China to take more of an interest in the country, but the Chinese have refused, preferring countries that offer five or six years of political stability.

One of the most valuable sections of Losing Afghanistan deals with the future of liberal interventionism. The 2001 invasion was often regarded as a reasonably good argument in favour of stepping in to rescue a country from disaster. But authorities as varied as Prof Paul Dixon, Prof Stephen Gethins, the writer Mahmud Khalili, Jeremy Purvis of the Liberal Democrats, Masoud Andarabi, interior minister of Afghanistan immediately before the Taliban takeover, and the book’s compiler, the distinguished historian Brian Brivati, make clear how and why August 2021 has changed all that. The era of Blair, Clinton and Bush is definitively over.

Victory for the Taliban wasn’t inevitable, but with hindsight there was too much corruption under the old regime, and the Afghan National Army was nowhere near as well trained and confident as the west assumed. The new regime has turned out to be more savvy than the old Taliban leadership of 1996-2001, under the reclusive Mullah Omar. Whereas the old Taliban blew up girls’ schools and murdered their pupils, the new Taliban have so far refused to issue any nationwide guidance about women’s education. Presumably they expect this will form part of a long-term deal with western powers over the country’s frozen financial assets.

At present, the Taliban control the cities and the surrounding countryside, but it’s only a matter of time before large-scale resistance gets going. If the Taliban hope to survive, they will have to be more inclusive, and accept a wider range of political and religious opinion. But old habits die hard, and the toll on individual lives will continue to be great. Losing Afghanistan is interleaved with brief but shocking accounts of the way people have suffered since the Taliban took over: “A policewoman faces a dangerous future”; “Losing my culture overnight”; “A mother turns to sex work”. This book provides a really valuable snapshot of the mess that President Biden chose to create, in a country that was entirely dependent on his whims.

One of the results of the chaos left by the western allies will, of course, be a renewed torrent of migrants to the outside world. In a moving and beautifully written book, The Naked Don’t Fear the Water, the respected journalist Matthieu Aikins, who writes for the New York Times Magazine and Rolling Stone, tells the story of a young Afghan driver and translator who decides to flee his war-torn country and make the perilous journey to the west. Omar escaped from Afghanistan in 2016, long before the Taliban took power again, but his reasons for leaving had long been present: economic hardship; a lack of freedom of expression that young, open-minded people find stifling; the desire to see and experience another world. Highly readable, empathetic and revealing, Aikins’s book is brutally honest and often deeply moving – a work of great sympathy and understanding. Above all, it explains why so many thousands of Afghans and others risk their lives to follow Omar’s lead, often with tragic consequences.

Now, more and more of them will be prepared to take their chances. Every day outside the Iranian consulates in cities such as Herat, thousands queue up in the hope of getting a visa. Few of them actually want to stay in Iran: they hope to use it as a staging point before moving westwards. The catastrophe that President Biden has inflicted on Afghanistan will have consequences in the region and the world for decades to come. And it was all done in a moment of carelessness, with no proper thought or understanding. As one former Afghan politician says: “It was utterly unforgivable. The Americans just didn’t care enough about our country to know what they were doing to us.”

John Simpson is the BBC’s world affairs editor

• Losing Afghanistan: The Fall of Kabul and the End of Western Intervention is published by Biteback Publishing (£20); The Naked Don’t Fear the Water is published by Fitzcarraldo Editions (£12.99). To support the Guardian and Observer order your copies at guardianbookshop.com. Delivery charges may apply