“What is truth, said jesting Pilate – and would not stay for an answer.” Thus philosopher Francis Bacon dramatised the opening of his famous Of Truth essay. Every human being alive, he thought, was prey to the bewitching temptation to disregard truth even to the point of deceiving ourselves, and to believe what it pleased us to believe – certainly disregarding truths put to us by others. In Pilate’s view, both Jesus Christ and the high priests urging his crucifixion were no more than partisans of their own particular truths – just like all of us. But, writing more than 400 years ago, Bacon thought that was not good enough. It was crucial, he said, to be honest and have shared truths, and for that he looked to the methodologies being pioneered by science. There were no partisan truths in nature – only immutable laws and facts awaiting discovery.

Today the temptation to believe our own truth to the point of collectively deceiving ourselves, hugely intensified by social media and its accompanying polarisation, has become an everyday talking point – the enemy of democracy and, arguably, civilisation itself. Bacon may have had faith in scientific facts to be the foundations of truth, but alarming proportions of educated people in western democracies do not share his faith even in science, distrusting what it has to say about everything from climate change to vaccination. If a message, especially political or cultural, is uncongenial, retreat to your own truth. Americans approach the 2024 presidential election with mounting foreboding that it will be characterised by industrial-scale misinformation – and there are parallel fears in Britain. Winston Churchill once said a lie can be halfway around the world before the truth can get its pants on – and that was before social media. False stories on X are 70% more likely to be retweeted than true ones – and true stories take six times longer to reach a sample of 1,500 than false ones.

There are a host of proposals in response, usually involving some mix of tougher policing of content by regulators and social media platform providers, supplemented by the creation of a public service digital platform committed to the posting of factual information. But the appetite for change is as yet hardly a tidal wave – and there is much more we as individuals can do to identify lies and facts bent out of kilter to support the advocacy of whatever cause.

Enter Alex Edmans. May Contain Lies is a wonderful litany of the myriad ways in which we can be deceived, and deceive ourselves, including sometimes well-known academic researchers as they try to stand up their theory. There are no sacred cows for Edmans. Whether it’s the authors of 2009’s famous The Spirit Level, which purported to show that inequality drives bad health outcomes, or the 1994 business book Built to Last, which influenced a generation with its apparent proof that visionary companies outlast their non-visionary peers, Edmans is unsparing. To test the famous result that it is inequality rather than poverty that causes bad health outcomes, you must statistically remove poverty from the data and run rigorous comparisons: when you do, says Edmans, you discover that the relationship is much weaker. Poverty remains a formidable cause of bad health. Equally the authors of Built to Last were pulling off the oldest trick in the book – selecting data that proved their point, in this case a collection of companies they knew were enduring, then retrofitting their character to prove their point. Small wonder so many of those visionary businesses have collapsed since the book’s publication.

Edmans is no less hard on himself. He tells the story of how he repeated for some years in his business school lectures the great Malcolm Gladwell statement that perfection requires 10,000 hours of practise. Then he inspected the data behind the statement and found that almost nothing did. (Here I should declare an interest: I know Edmans personally through our work on the Purposeful Company business thinktank and have experienced first-hand the full force of his demands for rigour. He practises what he preaches.)

So it goes on – the unconscious bias, the deceptive quest for statistics and facts that support your pre-rehearsed position, the exacting standards that must be met to prove a thesis and which so rarely are, and the dangers of “groupthink”. Edmans tells an engaging story of how President Kennedy, after the fiasco of the attempted invasion of Cuba in the Bay of Pigs, was determined not to make the same mistake again. So when 18 months later the Soviet Union started sending nuclear missiles to Cuba, he established as wide a group of advisers as possible to examine all the possible options – and not just the military’s favoured one of launching airstrikes. He went for a blockade, secured Khrushchev’s agreement to withdraw on the promise that the US would not invade Cuba, and nuclear war was averted. The more diverse a group you have deliberating whatever decision, the more likely you are to come to the right answer. Thus we escaped a third world war.

Given that we are increasingly engulfed by a sea of misinformation and bad public policy informed by self-deception, Edmans’s message could hardly be more timely. He urges us to follow Aristotle’s maxim: it is the mark of an educated mind to entertain a thought without necessarily accepting it. His advice is to stay open to the notion that you may be wrong, because you find the truth by testing your ideas against those who think differently. Always beware anyone who says they have found the holy grail in any field; they usually haven’t. Look out for the credentials of those spinning whatever line: have they an incentive not to be truth-tellers? Journalists, be vigilant about eye-catching research: it will have been released to achieve impact, so check that it is sound. And social media users, pause and reflect before you retweet. Are you sure this information is right? If we are all more wary, it may start to make a difference.

This Time No Mistakes: How to Remake Britain by Will Hutton is out now (Apollo, £25)

• May Contain Lies: How Stories, Statistics and Studies Exploit Our Biases – and What We Can Do About It by Alex Edmans is published by Penguin Business (£18.99). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply