The 1970s were “failure time for Britain”. The words are those of an astute adman who devised the Austin Metro launch of 1980, a brouhaha of flag-waving, commercialised nationalism that caught the hankering to end the defeats, discomforts and grievances of the tawdry, stop-go 1970s. Andy Beckett’s new book – studded with revealing interviews and thoughtful profiles from the early 1980s – evokes and explains the Austin Metro moment in 20th-century history. Politicians, capitalists, socialists, film-makers, peace activists, royalists and rioters were groping their way towards a new version of England; the truth is that Scotland does not matter much in Beckett’s account and Northern Ireland was a different story. He shows how an exasperated population itched for institutional change.

In addition to the Austin Metro publicity, there was the upbeat revivalism of Diana Spencer’s wedding to Prince Charles, England’s trouncing of Australia in the Headingley Test match of 1981, and the nostalgic, heart-stopping thrills of Chariots of Fire. This film about rival Olympic runners, exploring the sacrifices and self-fulfilment involved in competitive striving, and celebrating national pride in individual achievement, would have been scorned by previous generations of cinema-goers, its producer, Lord Puttnam, tells Beckett. But early 1980s audiences were sick of the drab, failed communitarianism of previous years and greeted this epoch-defining film with tears and cheers.

The Thatcher government responded to this temper rather than leading it. Until the Falklands war, her government lurched defensively from one crisis to another. In Thatcher’s early years of power, the loathing of her monetarist policies and personality shook her. In assessing the economic and industrial context for monetarism, Beckett is mild about trade union militancy and restrictive practices and underplays the extent to which activists saw workplace strife as a means to weaken and destabilise government institutions.

Beckett is a lucid, focussed writer who never digresses or indulges pet fixations. There is a wry, shrewd humanity to his historical interests that throws up constant parallels with 2015, but he never hectors his readers on what to think. He covers such subjects as the glossy consumerism of World of Interiors, The Official Sloane Ranger Handbook, the launch of Channel 4, the enterprise allowance scheme to foster small businesses, the London Docklands Development Corporation, the Limehouse declaration that launched the SDP (the first party to accept donations by credit card), the Iranian embassy siege, Northern Irish terrorism on the mainland, expectations of nuclear apocalypse, the emotional impact and social influence of Raymond Briggs’s novel When the Wind Blows, the deployment of 160 cruise missiles at Greenham Common and the women’s peace camp nearby.

Beckett’s discredits many glib assumptions in his section on social housing and Thatcher’s right-to-buy policy. Labour governments of the 1970s built progressively fewer council homes, not only because money was short, but because key figures doubted that renting from councils, often for life, was satisfying for tenants and that building programmes were economically sustainable. A Labour government report of 1977 accepted that “owning one’s own house is a basic and natural desire” for most people. Contrary to myth, social housing was not a dystopia of brutalist concrete towers, but predominantly semi-detached houses, with gardens, on the edge of towns, isolated and lacking services. Average English council house rent in 1981 was less than 7% of average income. Repairs and maintenance by council workers were often lamentable; the paradox is that strong trade unionism among public sector workers damaged the council housing stock.

The most resonant sections of this book concern urban politics in Liverpool and London. Beckett gives a compelling analysis of the Toxteth riots, which were the product of racism as much as economic deprivation. Discrimination meant that 70-80% of young black people were unemployed in the Liverpool 8 area. There were four black policemen among a Merseyside force of 5,000 and its bigotry was repulsive. Kenneth Oxford, the chief constable of Merseyside, liked to sound off on “the problem of half-castes” – “Negroes will not accept them as blacks and whites just assume they are coloureds.” Under his leadership, the unlawful behaviour, framing, ignorance, violence, obscenities and racism of the Liverpool police was horrific. Jimi Jagne, with a Chinese mother and a father from Gambia, is one of the most attractive voices among Beckett’s interviewees. The longest single quote in Promised You a Miracle is Jagne’s account of being stopped by police on his way home from school, abused, manhandled and having his schoolbag emptied into a wasteland puddle. Yet Thatcher was “amazed” by the Toxteth community’s hatred of the police.

It is Ken Livingstone rather than Margaret Thatcher who emerges as the most exciting harbinger of change. When this fearless and politically adept innovator became Labour leader of the GLC in 1981, it was the first time that anyone of the post-1945 generation had attained political power. Mocked by Max Hastings for being a man who did his own ironing, he was vilified for speaking the truth that Northern Irish violence would continue until there was a political settlement, which required negotiation with the IRA. His cut-price bus and underground fares scheme of 1981 brought an additional half-a-million people to public transport and was the first blow against the destructive primacy given to private motoring. Beckett provides a wonderful chapter on the GLC women’s committee promoting equal opportunities and opposing discrimination on grounds of gender. Nothing else in politics excited such rank-and-file participation, recognized potential in unsuspected people and places, and inaugurated identity politics. There is an unreasoning anti-London temper in leftwing politics today, so it is salutary to remember how much that is fundamental to our notions of good behaviour, respect, neighbourliness and civilised values come from the political leadership of the capital in the early 1980s.

Promised You a Miracle by Andy Beckett (Allan Lane, £20). To order a copy for £16, go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.