Although of modest origins, Chris Patten has led a golden life. Born to a lower-middle-class family in Ealing, he passed every exam he ever sat with flying colours, won every scholarship and was in due course elected Tory MP for the beautiful city of Bath. Before long he was writing speeches for the prime minister, then gradually rose through the ranks to the point where, by the election of 1992, he was chairman of the Conservative party, appearing on television every night. What could possibly go wrong?

It was at this point that he hit a bump in the road that irrevocably changed the course of his stellar political career. The Conservatives, led by his friend John Major, won the election, but Patten lost his seat. Even now, he was not short of options. He could easily have been wafted back into the cabinet via either a byelection in a safe seat (Kensington and Chelsea was mentioned) or elevation to the Lords. But there was another intriguing possibility – the governorship of Hong Kong. That is what he chose.

Patten has not set out to write a conventional memoir, though in many respects this is. Each episode from his varied career is used as a peg on which to hang his thoughts on the issues of the day. Thus we have chapters on Ireland, where he was a minister in the 80s, China (his time in Hong Kong), Europe (he was a EU commissioner), and so on. At the end there is a catch-all chapter, taking in his various other incarnations – chairman of the BBC Trust, chancellor of the Universities of Oxford and Newcastle. At one point, he was even enlisted by the Vatican to propose reform to its sclerotic bureaucracy. This is a man who has had his fingers in many pies. Not for nothing was he labelled by sketch writers the Grand Poobah.

Certain themes run throughout. Although he got on well with Margaret Thatcher, Patten, a practising Catholic, was an incorrigible “wet”. “I am a Conservative who doesn’t believe everything his party does is right.” He has the great merit of having held broadly consistent views throughout his life. His opinions are moderate and balanced. He is suspicious of zeal. His political hero is that epitome of one-nation conservatism, Rab Butler, who, says the author, “managed to find the right balance between expediency and Conservative principles”. He quotes with approval Butler’s disdain for austerity economics. “Those who talk about creating pools of unemployment should be thrown into them and made to swim.”

Of the three Conservative leaders with whom he worked – Heath, Thatcher and Major – Major comes off best. “John was one of the most decent people ever to lead the Conservative party… He actually listened to other opinions… But its senior members had got used to being roughed up, and seemed to find it difficult to summon the grace to behave well when they were themselves treated like grown ups.”

Although a lifelong pro-European and, therefore, not someone who would have thrived in the modern Tory party, Patten is not uncritical of the EU. “A besetting sin in the EU… is to duck hard questions in order to secure soft answers.” Thus, in its haste to absorb the former east European states, the EU turned a blind eye to their manifest unreadiness. The launch of the single currency was fundamentally flawed. “It was all very well,” says the author, “scolding the Greeks for being too idle, too corrupt, the French for being too dependent on a bloated public sector, the Portuguese for being too far down any queue of economic comparators. That might all be true. But for sure you could not wave a wand or offer a blessing and save these sinners overnight. They each sinned in their own chosen way and, however tiresome that may be, you could not make them all into Germans.”

What does he make of the present mess? “The Brexit vote in the UK and the election of President Trump in the USA have together threatened to destroy the foundations of the world order to which my political life, at home and abroad, has been devoted.” He is in no doubt about where the blame lies. “It would be unreasonable to look beyond the Conservative party. Attempts to blame Labour and Mr Corbyn are far-fetched. About two-thirds of those who voted Labour in the 2015 general election voted to remain in the EU; less than half of Conservatives did so. The 2017 election was above all another attempt to manage the Conservative party.” Amen.

Chris Mullin is a former Labour minister. His recent memoir, Hinterland, is published by Profile Books (£20).

• First Confession: A Sort of Memoir by Chris Patten is published by Allen Lane (£20). To order a copy for £15 go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99