“Few members have yet tumbled to the juggernaut heading their way,” remarked the late Robin Cook when the Freedom of Information Act came into force, which, among other things, required the publication of MPs’ expenses three years in arrears.

How right he was. It was another six years before the bombshell exploded, during which time the Commons authorities did everything they could to prevent publication. At one point, the whips on both sides conspired unsuccessfully to wave through an amendment exempting parliament from the provisions of the act. In the end, their hands were forced when the Daily Telegraph purchased a stolen copy of the computer disc containing all the details and began drip-feeding them in detail over a period of weeks.

And what a story it was. Replete with tales of duck houses, moat clearing, wisteria trimming, champagne flutes, bath plugs, all purchased with public money. There were even cases of outright fraud and in due course prosecutions followed. Several MPs were eventually jailed and many fell on their swords, announcing early retirement.

The impact on the political classes was traumatic. No matter that many MPs were not implicated (I was more or less unscathed, being well down the league table of claimants), we were all tarred with the same brush. In the red-top tabloids, the narrative quickly became “they’re all at it”; the damage to public confidence in politics and politicians was incalculable. Arguably, it contributed to the mood of populist outrage that led to Brexit.

Political leaders moved quickly to damp down the outrage. Inquiries were announced; rules about what could and could not be claimed were hastily revised; root-and-branch reform was promised. This led in due course to the creation of the Independent Parliamentary Standards Authority (Ipsa), to be headed by Professor Sir Ian Kennedy, a distinguished public servant with long experience of regulating the public sector, but nothing in Kennedy’s experience prepared him for what followed.

Cleaning Up the Mess is his account of setting up and running Ipsa. The idea was that it would be independent of parliament and entirely transparent, freeing MPs once and for all from the embarrassing responsibility for setting their own terms and conditions. In due course, it would also adjudicate on MPs’ pay and pensions. However, despite having overwhelmingly endorsed the concept, once Ipsa was up and running many MPs began to have second thoughts.

As Kennedy says, he was dealing with a group of people who were not used to the idea of regulation and who on the whole resented it. “It mattered little that the system was familiar to anyone who enjoyed the benefit of other people’s money and understood that expenditure had to be accounted for. It mattered even less that the idea of creating a regulatory system independent of parliament was their idea, one that had been voted for by parliament itself.”

To be fair, Ipsa got off to a rocky start. There were teething troubles with computers and many MPs and their staff struggled with a system that was overly bureaucratic, but nothing excused the misbehaviour of a significant minority of MPs. Years of laxity had given rise to a sense of entitlement that proved difficult to erase. By and large, the new generation of MPs coped, but the old lags were the worst. Kennedy and his staff were abused, bullied, ranted at. There were multiple specious parliamentary inquiries into the workings of Ipsa, the aim of which seemed to be to derail the whole process.

The media were a problem, too. Many journalists were hooked on the “all MPs are crooks” agenda and reluctant to let go even though, as Kennedy makes clear, this was not the case. In the long run, however, the story, though not the memory, faded. Effective regulation means there are no more abuses. Transparency means there are no more scandals to disinter. Ipsa, as Kennedy says, “had effectively recalibrated the relationship between taxpayers and MPs”. Far from resenting it, MPs should be grateful.

Chris Mullin was MP for Sunderland South from 1987-2010. His latest novel, The Friends of Harry Perkins, is published by Scribner (£12.99).

• Cleaning Up the Mess by Ian Kennedy is published by Biteback (£20). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p on all online orders over £15. Phone orders min p&p of £1.99