Like many anguished teens, I often felt that my best friend was my diary. I would enter my bedroom in a terrible mood, but as the sentences took shape on the page, whatever was troubling me no longer felt like quite as much of a catastrophe. I wasn’t able to extinguish every sadness, but often felt calmer, as if a physical pressure had lifted from my chest.

These moments always brought to mind a scene in Harry Potter and the Goblet of Fire in which the headteacher at Hogwarts, Albus Dumbledore, extracts upsetting memories with his magic wand and places them in a shallow bowl, called the “pensieve”, which allows him to view things more dispassionately. Writing, for me, provided the same relief.

Diarists throughout history have experienced something similar. “When I write I can shake off all my cares. My sorrow disappears, my spirits are revived,” Anne Frank observed. As a science writer, I have been delighted to discover that the practice doesn’t only soothe a troubled soul, it can even boost our physical health.

Professor James Pennebaker and his graduate student Sandra Beall were the first to demonstrate these effects in the 1980s. They asked a series of students to write a short essay for 15 minutes on four consecutive days. Some of the participants were encouraged to write about “the most traumatic or upsetting experience” of their lives, exploring their “deepest thoughts and feelings” about the episode. Others were asked to write about trivialities, such as descriptions of their dorm rooms or the shoes they were wearing.

The students in the first group described feelings of guilt over a grandmother’s death, anger over their parents’ divorce and torment about their sexuality. Confronting their feelings was not easy, and many left Pennebaker’s lab feeling sadder than when they came in. Over the following six months, however, they paid around half as many visits to the student health centre as the participants in the control group.

This may sound miraculous, but emotional distress of all kinds can throw the immune system out of whack, meaning that it responds less efficiently to infection. If expressive writing ultimately leaves us feeling calmer, then it could restore those defences to their full strength. And that is exactly what Pennebaker’s colleagues revealed using blood tests: after writing about their feelings for four consecutive days, people’s white blood cells started replicating more quickly in response to foreign invaders. Subsequent studies have found that expressive writing can accelerate recovery after a biopsy. It also seems to reduce blood pressure and improve lung function.

The link between stress and illness may be clear, but why is writing so cathartic in the first place? That’s trickier to answer. One possibility is that externalising our thoughts gives us more head space to think about other things. We now know that simply writing a to-do list can release people’s cognitive resources for other activities, as it reduces the amount of information they are juggling in their minds. This eases stress and, done before bedtime, it can even improve sleep.

Expressive writing could perform a similar function in clearing our mental workspace of the sources of negative rumination, freeing up working memory to devote to the things that matter. That chimes with my own experience. When I feel distressed, my anxieties sit centre stage in my mind and prevent me from focusing on pleasant activities such as reading. After I have written in my journal, these demons recede into the background, having been given the attention they crave.

Equally importantly, writing our worries down can create a sense of “psychological distance”, which allows us to take a more philosophical attitude to our problems. If we have been contemplating hurt caused by others, for instance, we might find it easier to see their viewpoint, or we recognise a valuable lesson we’ve learned, which could help us to function better in the future.

In some cases we might even be able to see the funny side of the situation – or at least take ownership of what happened. “When you slip on a banana peel, people laugh at you; but when you tell people you slipped on a banana peel, it’s your laugh,” wrote Nora Ephron. Expressive writing can help us to make this mental transformation in private.

No single intervention will suit everyone: writing is just one tool and some may balk at the idea of setting out their feelings in detail. If you would like to incorporate it into your own mental first aid kit, though, there are some evidence-based ways to maximise its benefits.

Without losing too much spontaneity, aim to give your account a narrative structure, such as sketching out the background to the situation and the chain of events that led up to it. Be precise about the emotions you are experiencing: is it irritation or frustration; disappointment or disillusionment? The act of pinpointing those exact feelings can make insight easier. Finally, consider how your situation affirms your good qualities and the things you value. You might be demonstrating strength, or kindness, or you might become newly aware of the importance of certain friendships in your life. If I am particularly anxious, I try to frame the diary entry as a letter from a friend who is offering kind advice. This imaginative exercise boosts self-compassion, which reduces distress. I also find it helpful to note down simple moments of pleasure.

My teenage angst may be well behind me, but my journal continues to be a source of solace – and with my knowledge of the psychological literature, I can now use it to full advantage. Whenever I feel overwhelmed, a few minutes with my pen in my hand makes the world feel a little more manageable.

• David Robson is the author of The Expectation Effect: How Your Mindset Can Transform Your Life (Canongate).

Further reading

Opening Up by Writing It Down by James W Pennebaker and Joshua M Smyth (Guilford, £14.99)

The Extended Mind: The Power of Thinking Outside the Brain by Annie Murphy Paul (Mariner, £28)

Write It All Down: How to Put Your Life on the Page by Cathy Rentzenbrink (Bluebird, £14.99)