There are few genres with a hold on the public imagination like spy fiction. This year especially it seems that spies are everywhere, whether in the form of Kit Harington brooding his way through the gritty, Bourne-style movie Spooks: The Greater Good, Daniel Craig’s once-more dependable 007 in the forthcoming Spectre, or played for laughs in Spy by Melissa McCarthy, Miranda Hart and Jason Statham. This week the new James Bond novel is published – entitled Trigger Mortis and written by Anthony Horowitz in the style of Ian Fleming (see review on page 11): the latest in a line of nostalgic additions to the Bond canon after Sebastian Faulks, Jeffery Deaver and William Boyd have already tried their luck.

This popularity is, of course, nothing new. The figure of the spy has lurked in the background of all types of literature for a couple of millennia, but it was around the turn of the 20th century that spy fiction became more closely associated in Britain with nation and empire, and grew into a commercially successful genre in its own right. Against a backdrop of public anxiety over Anglo-German tensions in Europe, espionage novels such as Erskine Childers’s The Riddle of the Sands (1903) gripped the nation and helped launch the careers of William Le Queux and, later, John Buchan. They joined other apparently more serious authors such as Rudyard Kipling, whose Kim (1901) fictionalised the “Great Game” of espionage in colonial India; and Joseph Conrad, who explored the underbelly of fin-de‑siècle London in The Secret Agent (1907). In the interwar years Eric Ambler, Manning Coles, Graham Greene and others continued to develop many of spy fiction’s conventions. It was in the aftermath of the second world war, however, that the genre made its transition from national to global phenomenon.

The British empire, victorious but worn out by the war, had begun its long process of dissolution, starting with Indian independence in 1947 and ending (eventually) with the return of Hong Kong to China in 1997. Spy fiction not only spoke to anxieties over the emergent cold war, but also mitigated the fear of international decline in an era of swingeing cuts to British armed forces and costly conflicts in Malaya, Kenya, Aden and Suez, suggesting to the reading public that a vanishing empire was never more secure. Spies such as Fleming’s Bond in particular satisfied the need for reassurance that Britain was still as much of a player on the global stage as it had ever been. As Bond tells his contact in Japanese intelligence, Tiger Tanaka, in You Only Live Twice: “England may have been bled pretty thin by a couple of world wars, our welfare state politics may have made us expect too much for free, and the liberation of our colonies may have gone too fast, but we still climb Everest and beat plenty of the world at plenty of sports and win Nobel prizes.”

Bond proved quite convincing. So successful was Fleming’s vision of the spy, and so dominant has Bond been in popular culture over the past 60 years, that it is near-impossible to discuss cold war fiction without mentioning him. The first of Fleming’s novels, Casino Royale, was published with modest success in 1953. But over the course of the next decade, his books, published at a rate of one a year until his death in 1964, began to be serialised in newspapers and magazines before expanding into radio and film. The Broccoli-Saltzmann productions of the Bond series began in 1962 with Dr No. Connery’s portrayal not only consolidated the image of the spy in popular media, but also made Bond synonymous with ideas of Britishness and nation (who can forget his prominent role at the London 2012 Olympics opening ceremony?).

Fleming was always broadly conservative in his political outlook (his father, Valentine Fleming, was a Conservative MP and friends with Winston Churchill) and both the novel and film versions of Dr No offer insight into his opinions on decolonisation and the cold war politics that drove it. Fleming owned a home in Jamaica, named Goldeneye, which he lent to Anthony Eden after Suez so that he could recover away from the media spotlight. Fleming’s representation of the Caribbean reflected government policy throughout the 1940s and 50s, suggesting that the British position was best served by acting as a strategic partner to the US, with the maintenance of colonial possessions as an essential buffer zone against Soviet expansion. His intent in the novel is to illustrate that Britain can be an equal ally in the struggle against communism, while maintaining direct control over a colonial empire.

In the novel, Dr No is reputed to be disrupting American intercontinental ballistic missile tests conducted in the Caribbean. In the film, he instead threatens rocket launches from Cape Canaveral and the activities of Nasa’s recently inaugurated Apollo programme, key to winning the space race. In both cases, Bond is dispatched to stop him. In the closing scenes of the novel, Bond sits down to dinner with Honeychile Rider, who, it transpires, is the heiress to an interregnum-era plantation outside of Kingston. Past and present are linked: the postwar special relationship and the age of colonial expansion into the West Indies. There is no hint of British decline.

Fleming’s contemporary Graham Greene, and, later, John le Carré, possessed a firmer grasp of realpolitik, both with regard to the state of the empire and the international pecking order. The ageing reporter Thomas Fowler in Greene’s The Quiet American (1955) realises, to echo Harold Macmillan’s words, that the British were the declining “Greeks” to the American “Romans”. British characters retain their relevance, but not without acknowledging their downgraded status. Few of them can resist sniping at the Americans from the sidelines, and there is a strong anti‑American streak that runs through the genre in Britain. Greene’s Fowler eventually turns against CIA operative Alden Pyle, ostensibly over a botched car-bombing but actually because of Pyle’s involvement with his lover Phuong; while in Le Carré’s The Honourable Schoolboy (1977), another journalist and intelligence agent, Jerry Westerby, takes a perverse pleasure in the despondency of American personnel at an air force base in Cambodia after the fall of Saigon, seeing it as proof that the informal US empire was no match for the British. Both Fowler and Westerby’s attitudes are rooted in the British envy of a former colony now in global ascendancy; both characters swing between weary sagacity and open disdain for their US counterparts.

Other writers acknowledged geopolitical shifts more obliquely. Len Deighton’s novels all but avoid the former territories of empire, instead focusing on a rapidly changing London, or the new arenas of cold war conflict such as Beirut and western Europe. Whereas Bond is squarely aligned with the establishment he serves, Deighton’s unnamed spy from The Ipcress File (1962) prefers the bohemian fringes of contemporary life, from his taste for coffee houses in London’s Soho to an interest in gourmet cookery, one of the author’s own passions. The empire is omitted. This decision had become more plausible, perhaps, by the early 1960s, when Bond, the Beatles and the “British invasion” made up for the nation’s political shortcomings. A decade later, Le Carré’s Tinker Tailor Soldier Spy (1974) would confront far more directly the loss of empire and the degraded fortunes of Britain. A tense and melancholy novel, its establishment characters represent a generation of men who, like Bond, were “trained to empire, trained to rule the waves”, but are now left adrift and redundant in a postcolonial world.

Half a century of Bond films later, such ideas of decline appear to have been forgotten. The film budgets have swollen and the book launch events grown more spectacular, but many of the imperial undercurrents of spy fiction remain. Skyfall is littered with the imagery of empire and nation, all union jacks and bulldogs, and the antagonist Raoul Silva’s villainous tendencies are explained away as a result of his having spent too long in the east. Boyd’s Solo (2013) also rearticulated spy fiction’s colonial past, with Bond going undercover as a journalist in the midst of an African civil war. Set in 1969, the novel fictionalises the Biafra-Nigeria conflict of 1967-70 with evident postcolonial overtones – M’s lament over the passing of the “dominant and impressive” British empire in Africa is just one example. (The novel chimes with this week’s revelation that another celebrated spy writer, Frederick Forsyth, reported to MI6 from Biafra while working as a freelance journalist for various magazines and newspapers. Forsyth’s recruitment in 1968 began a relationship between the writer and the intelligence service that apparently lasted more than 20 years. Like many of his contemporaries, he demonstrates how the line between fiction and reality in espionage is often a hazy one.)

What’s more, the racial tensions inherent in the Bond series have resurfaced. Earlier this year, there was a brief social media storm over Roger Moore’s alleged disapproval of Idris Elba’s potential casting as the next Bond; and this week Horowitz has had to apologise for his comment that Elba would be too “street” to play 007.

Many of the political concerns that influenced cold war spy fiction continue to resonate. Britain still seeks to punch above its weight in the international arena, despite the nation’s dwindling armed forces. And stories about espionage remain as marketable as ever.

• Sam Goodman is the author of British Spy Fiction and the End of Empire.