As Halloween approaches, we prepare to confront our ghosts. Soon we’ll be used to ghoulish children leaping out of shadows in the street. Meanwhile in churches, for the three days of Allhallowtide, Christians will remember the faithful dead.

But ghosts aren’t always as recognisable as they are now in the guise of trick-or-treaters. “Ghosts have grown up,” Elizabeth Bowen wrote in a preface to a new book of ghost stories in 1952. They had laid aside their original bag of tricks – “bleeding hands, luminous skulls and so on” – and were now more likely to be found in a prosaic scene. “Today’s haunted room has a rosy wallpaper.” Most frighteningly, “contemporary ghosts are credible”. They lurked at the border of known reality, just believable enough to unnerve those who encountered them in life or art.

In fact, the growing up that Bowen describes began much earlier. Ghosts in Britain have a long history since Grendel crashed through the door in Beowulf, eating the warriors who get in his way, and Susan Owens has set out to tell it in an eloquent and lively account. For Owens, ghosts – and especially their appearances in art and literature – offer a window on to “the great changes that, over time, have made us see the world in new ways”. The Reformation, the Enlightenment, the age of technology – all have shaped the development of ghosts, and look different when seen through a ghostly lens.

According to Owens, ghosts have performed two functions. The first is to scare us, reminding us of the presence of death. The second is to reassure us, promising that death may not be as final a state as it seems. In the 14th and 15th centuries, they dwelt in purgatory, hovering between heaven and hell, and so were able to warn the living of the dangers of sin at the same time as offering the promise of eventual redemption. This changed with the Reformation, when purgatory was officially abolished. The clergyman Robert Wisdom lamented in 1543 that “sowles departed do not come again and play boo peape with us”. But like so many beliefs, the notion of purgatory lingered in the popular imagination long afterwards. Thus in 1609 the ghost of Hamlet’s father informed his son that he was “doomed for a certain term to walk the night”, though he didn’t actually name it as purgatory.

It was in the 17th century that the modernisation of ghosts described by Bowen really began. John Donne was an important figure here. “Then shall my ghost come to thy bed,” he informs a recalcitrant lover in The Apparition, threatening to scare her so much that she’ll end as “A verier ghost than I”. This is, as Owens observes, the ghost as a “physical passionate being” of flesh and blood, capable of loving after death.

The ghost was threatened by the Enlightenment but saved by the prevailing melancholia of the Romantic era. For Shelley, in particular, ghosts were symbols of natural renewal; it was reassuring to think that dead humans contained sparks of life as dead leaves did. These imaginative spirits had their more energetic heirs in the phantasmagorias of the Victorian age. Suddenly ghosts could be captured on screens and in photographs. Indeed the medium of photography, with its pallid figures emerging out of shadowy backgrounds, seemed to make ghosts of all its subjects. This was also the era of spiritualism – when mediums allowed matter-of-fact contact with the dead – only weakly combated by the Society for Psychical Research, founded in 1882 to provide rational explanations for apparently otherworldly phenomena.

And so here we are now, in an age of sceptical belief. “All argument is against it; but all belief is for it,” Dr Johnson announced in the 18th century, and I agree with Owens that not much has fundamentally changed since then. She admits to a “balance of scepticism and credulity that would probably not stand up to rigorous scrutiny”. I suspect that many of her readers will share this viewpoint, dismissing the claims of the supernatural while remaining on uneasily familiar terms with the family ghost. We want to have it both ways, like atheists who resort to prayer on special occasions. According to Owen, this psychic hedging of bets may be an “intrinsic part of what it is to be human”.

There are some wonderful contemporary responses to the ghostly realm that prove Bowen’s pronouncement to be true. Owens is good on the artist Susan Hiller’s strange ghostly installations, manipulating light and sound on screens placed in domestic interiors as though operating as frontiers between the living and the dead. She quotes from Hilary Mantel’s 2005 Beyond Black – a brilliantly witty romp through the macabre world of suburban spiritualism. Ghosts here bounce around on the back seats of cars demanding to stop at service stations and pop up next to you in the bath. These are ghosts who make their presence idiosyncratically and tiresomely felt because, as one reminds the weary medium who serves as the book’s protagonist, “you don’t get a personality transplant when you’re dead”.

• The Ghost: A Cultural History by Susan Owens is published by Tate (£19.99). To order a copy for £16.99 go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99

• Lara Feigel is the author of The Bitter Taste of Victory: Life, Love and Art in the Ruins of the Reich (Bloomsbury)