Before it became a catch-all term for the embarrassing antics of middle-aged men, “midlife crisis” meant something specific: that moment, any time from about age 35, when the goals you’ve been pursuing in adulthood don’t seem worth it any more.

It’s bad enough if you’ve failed to achieve your career ambitions, win the respect of your peers or fashion a comfortable lifestyle. But in some ways it’s worse if you’ve succeeded, because then it’s all too obvious that such things don’t bring deep fulfilment.

In David Brooks’s governing metaphor, you’ve made it to the summit of life’s “first mountain”, only to discover that the view isn’t really so great and you feel empty inside. The truly joyful people are those, often impelled by a shock such as divorce or bereavement, who find their second mountain, abandoning themselves to a greater cause, forgoing the life they’d wanted for whatever the world needs from them.

In part, Brooks’s stirring new book is a map for this journey: a self-help guide to escaping the prison of self.

But this isn’t quite the heart of his outlook. Nor, for that matter, would it explain Brooks’s status as the favoured punchbag of a generation of younger American journalists, to whom the New York Times columnist and television pundit is an insufferable scold – “the biggest windbag in the western hemisphere”, in the words of Rolling Stone’s Matt Taibbi.

(Gawker, the deceased gossip blog, once described him with characteristic scorn as “basically the nice old man with Alzheimer’s at church that everyone chooses to leave alone as long as he doesn’t hit anyone”.)

His larger, and to some, vastly more irritating, argument is that this crisis of meaning is societal. The modern world is collectively stranded on the first mountain, a culture of radical individualism dividing us into “insecure underachievers” and left-behind no-hopers, the former only slightly happier than the latter, with tribal politics offering one of the only ways to feel a sense of belonging.

Brooks, a centrist conservative, surprised many recently by endorsing the case for slavery reparations, but seen through this moral lens, his position makes sense. If slaveholding was a sin, it demands expiation; to change its politics, a nation must attend to the stain on its soul.

The book builds its case through stories of people who took the key step toward the second mountain, which is commitment – to a spouse, community, faith or philosophy. “The world tells them to be a good consumer, but they want to be the one consumed – by a moral cause,” Brooks writes. “The world tells them to want independence, but they want interdependence.”

When his first child was born, a friend sent him an email – “welcome to the world of unavoidable reality” – and those he presents as paragons take that logic to its extreme. Thus, for example, we meet a couple who began providing occasional meals to a classmate of their son, who might otherwise go to bed hungry, a gesture that snowballed, “simply by responding to the needs around them”, into a weekly dinner for 25 kids, a home for several, plus an annual seaside holiday for as many as 40. Such “radical hospitality” doesn’t make for a life of freedom in the usual sense.

But it meets the theologian Tim Keller’s definition of a deeper kind of freedom: “Not so much the absence of restrictions as finding the right ones.”

The Second Mountain is a memoir, too, featuring passages that will provide red meat for Brooks’s media mockers, since they describe leaving his wife of 27 years for a much younger woman who’d worked as his researcher. (There’s also an ambivalent religious conversion, from Jewish to Christian-ish.) To his credit, he’s as lacerating about himself as any gossip blog.

“I was unplanted, lonely, humiliated, scattered,” he writes. “I was throwing myself needily upon my friends in ways that are embarrassing now if I stop to remember them, which I try not to… [I was] confronting the problems of a 22-year-old with the mind of a 52-year-old.”

Brooks has always included himself among his targets, ever since the excellent Bobos in Paradise (2000), which took confident aim at “bourgeois bohemians”, with their overpriced kitchen appliances and “distressed” clothes and furniture. (“Across the developing world, there are factory workers busy beating up the goods they have just made in order to please American consumers.”) But we’re in more personal territory now.

The obvious criticism here, as with much of Brooks’s work, is that his conservative temperament blinds him to the role of large-scale political reform in fixing the social ills that trouble him. The risk of all this talk of commitment and charity among neighbours is to give the impression that structural problems – economic inequality, racism – might vanish if we all just worked on our souls.

But while there’s merit to this objection, it’s also far too easy a refuge for those who’d rather not look too deeply within. Calling for political change can be much less scary than asking if you, yourself, might need to switch mountains.

Paraphrasing TS Eliot, Brooks writes that the chief illusion of modern politics is “that you can build a system so perfect that the people in it do not have to be good”. This powerful book, Brooks’s best to date, may be especially valuable to those convinced they don’t need it.

• The Second Mountain: The Quest for a Moral Life by David Brooks is published by Allen Lane (£20). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99