We are now in the early stages of a new phase of Islamic militancy. The death of Osama bin Laden in May 2011, and the outbreak of revolts across much of the Arab world, marked the end of a cycle that had commenced with the 9/11 attacks. This cycle had peaked in terms of violence around 2005 and then subsided towards the end of the decade as militant groups found their geographic and ideological space squeezed. It now looks like we are on another upswing with vast tracts of desert and dozens of towns under extremist authority in the Middle East and a new energy flowing through the myriad networks that make up the movement as a whole.

One can, however, identify four categories of militant activity at the moment. There are two main groups battling for pre-eminence: the veteran al-Qaida and the newcomers, the Islamic State or Daesh. There are various organisations affiliated to the former and loyal to the latter. Some are getting strong, some weaker, but most are proving remarkably tenacious. There are other groups that are entirely independent, though they may have some associative links with other militants, like the loathsome Boko Haram in Nigeria. And then there are the freelancers, the self-forming networks, the DIY terrorists who are increasingly responsible for violence on our streets. This picture is depressing, to say the least.

The capability of the militants to do harm comes from connections between groups, and particularly between these four categories. So of the three men who launched attacks in Paris in early January, one attacked a Jewish supermarket, killing four, while the other two gunned down 12 people, including 10 members of the editorial staff of the satirical magazine Charlie Hebdo. The former, in isolation, would have been a minor incident, albeit terrifying, and a powerful reminder of growing antisemitism in Europe. But the latter was of global significance, prompting massive commentary and attention from media and global leaders and, with its carefully chosen target, revealing deeply polarised attitudes between many in the west and many in the Islamic world. It was the Charlie Hebdo attack that had its origins in the visit of one of the killers to Yemen and his contact with an al-Qaida affiliate there.

In his intelligent, important new book, Patrick Cockburn concentrates on the role of the Syrian and Iraqi conflicts in this new landscape of jihad. As a reporter for the Independent, Cockburn has built up a well-deserved reputation for sensible, sober journalism, rooted in time spent on the ground as well as thinking and reading. He recently won a deserved award for spotting the rise of the Islamic State well before other observers. This most recent work is welcome. Amid the many books published on the current conflicts reshaping the Middle East, few are as informative or perceptive as The Rise of Islamic State.

The roots of the IS lie in the surge of violent Islamic activism in the Middle East of the 1980s and the effects of the war against the Soviets in Afghanistan, which brought a young Jordanian street thug known as Abu Musab al-Zarqawi to Afghanistan in 1989. He was too late to join the war but returned to his native land to plan attacks there. Jailed, al-Zarqawi was released in time to return to Afghanistan to create his own group, Tawhid wal-Jihad. His opportunity came with the US-led invasion of Iraq in 2003 and the subsequent uprising. Al-Zarqawi established himself as leader of the most brutal fringe of the insurgency. He was killed in 2006 as the sectarian civil war he had worked to foment intensified. If over the next four years the Islamic State in Iraq, as the group called itself, suffered under pressure from the US, it was able to regroup once the foreign troops had left. Under its new leader, Abu Bakr al-Baghdadi, the ISI launched new campaigns.

The 2011 revolt in Syria, and that country’s rapid disintegration into civil war, provided a new opportunity. Working with al-Qaida central, the ISI set up a new militant group in the neighbouring country. However, lines of command were never clear. Al-Baghdadi thought the new organisation was under his authority. Its commanders, and the al-Qaida command, thought differently. The result was an acrimonious split, al-Baghdadi sending forces to take over substantial portions of eastern Syria, while appropriating large chunks of a resurgent Iraqi Sunni insurgency against a Shia chauvinist government in Baghdad. By summer last year, al-Baghdadi was ready for a big push. He launched a successful attack on Mosul, Iraq’s troubled second city, and then declared himself caliph, temporal and spiritual ruler of the world’s Muslims.

So why, if people like Cockburn could see what was happening, did western security officials, analysts and editors miss it? Probably because, as The Rise of Islamic State explains, western policymakers have shown little but wishful thinking and inconsistency in dealing with the conflict in Syria or the supposed peace in Iraq for several years. Of all the many mistakes Cockburn says were made by both the rebels and their foreign backers since 2011, it was the belief that President Assad was going to be swiftly defeated that was the most serious. As late as 2012, foreign governments and journalists were speculating where al-Assad might choose for exile, despite holding, at the time, every one of Syria’s 14 provincial capitals. Since then al-Assad has lost just one, Raqqa, to the Islamic State. As for Iraq, no one wanted to believe how badly things were going. And with few journalists on the ground, internet rumour and the statements of interested politicians were all anyone had to go on.

Cockburn describes the civil war in Syria as “a Middle Eastern version of the 30 years war in Germany of the 17th century. All sides exaggerate their own strength and imagine that temporary success on the battlefield will open the way to total victory”. He refers to the “politics of the last atrocity”. The outcome probably rests with the US, Russia, Saudi Arabia and Iran, he says, all of whom have different interests and objectives.

One of the major themes that emerges from Cockburn’s account is the role of states in the whole appalling story. Islamic militant groups are usually described as non-state actors. But as The Rise of Islamic State makes clear, this is far from true. Everybody now seems to have some kind of involvement in this fight, which may have killed more than 200,000 people, and no one has a realistic idea of how to end it.

The Rise of Islamic State is published by Verso (£9.99). Click here to order it for £7.99