In 1977, when he was 16, Sam Miller’s mother, Jane, turned to him and said: “There’s something I want to tell you.” What followed was brief and to the point. Sam’s real father was not, she announced, her husband, the writer and legendary editor Karl Miller, but his close friend, Tony White, who had died, at the age of just 45, only the year before. Karl, she said, had known the truth about this right from the start, and had always loved Sam as his own. Mother and son then returned to the job in hand, which was the painting of his bedroom a somewhat forbidding shade of dark blue.

Family secrets are supposed to be corrosive. In a novel, such a revelation might have turned our hero overnight into a troubled teenager. But though he writes in his new memoir, Fathers, that his mother’s words “set off a slow-motion cascade of feelings and reactions and consequences”, most readers will be struck by the calmness with which Miller received them, and by how little real effect they had on his world. In their house in Chelsea, where Jane still lives today (and where I meet her son on a crisp February morning), life went on as usual. Karl was teaching at University College London, where he’d recently been appointed a professor of English literature; Jane, a future academic and writer, was busy training teachers; Sam and his siblings – Miller has an older brother and a younger sister – continued to shuttle between home and school. In 1979, Sam went to visit White’s cottage in Connemara and his (then) unmarked grave. But otherwise there were “no immediate consequences”. It was another 35 years before he and Karl would refer to the matter.

Fathers, then, is not a misery memoir. Far from it. It is, I suppose, a kind of detective story, its author seeking to piece together both the too-short life of the charismatic but rather elusive Tony White, a one-time actor who came to have an itinerant existence, with no permanent job and no permanent home, and the gilded early world of the Millers, with whom he had become friends at Cambridge in the 1950s (at university, Karl Miller, later the co-founder of the London Review of Books, was an Apostle who, as the editor of Granta, published early work by Ted Hughes and Thom Gunn). How, Miller seeks to discover, did his mother come to have an affair with White, and what effect did this have on Karl? After all, when it was over, the three of them remained close, and when White died, the Millers were “paralysed” with grief.

Mostly, though, it is a book about love: chiefly, the powerful love Sam and his father, Karl, felt for one another. “He was wonderful to me, and for me,” he insists. “We had a relationship that felt stronger and more meaningful than most father-son relationships.”

Where, then, does this leave the matter of blood? Fathers isn’t particularly interested in this territory. “That’s mainly a game, isn’t it? Those conventions: the cruel stepmother, the cinderella child, the changeling. They become self-fulfilling prophecies, ones that can probably cripple those who lack the confidence to see beyond them.” He, though, had that confidence: it passed to him from Karl, like a baton. “What does it [blood] matter? For me, I’m not sure it matters at all.” I don’t quite believe this. Since he won’t talk about White to me, the possibility exists that this may just be a way of putting the subject in a box and shoving it, metaphorically speaking, beneath a dusty bed.

Why did he write the book?

“It started as a story for my children, who were very close to my father. I read a version of that at his funeral, and then I just came back here, and sat in his study, and started my unofficial job of going through his books and papers. That’s when I began, and nothing I’ve written [Miller, a former BBC journalist, is the author of three books about India] has ever come so naturally. I was trying to sort out a few things: the circumstances of his death, and my feelings about that. We knew he was going to die: I should have been prepared. But I just wasn’t.”

Still, all this comes with a proviso: “When somebody dies, everyone wants to tell their story. I will have captured a lot of what other people in the family thought. But this is my account, not theirs.”

Has writing it helped him? Does he feel better, somehow?

“I’m bad at psychoanalysing myself. It may have helped, but maybe that’s also about time and the natural process of grieving.”

His memoir is written in tentative, floating, haiku-like paragraphs, from which Karl Miller rises indomitably: a romantic, adorable, half-mythologised figure (his son makes no mention, for instance, of his father’s capacity, as detailed in one obituary, for fallings out). Miller the younger sees Miller the elder’s capacity both for friendship and for family life – a family life he would not allow an affair to derail – as a choice, one that was a reaction to his own upbringing (Karl Miller’s parents having separated soon after he was born, he was raised by his maternal grandmother in Edinburgh). “Karl’s father was his opposite: a solitary man who went about with a sketchbook and Ryvita in his satchel, who had no friends. It’s almost like Karl took a decision about what he wanted to be. Growing up, there were no nuclear families in his vicinity. He didn’t have those norms – those sometimes very smug norms – about happy families and what they should consist of.”

Karl Miller was, in other words, all about other people: the writers and poets he nurtured and published; his many friends, including those, like Tony, with whom he played football; and his wife and children. He was, though this is not a word his son uses, a ringmaster.

Reading about his parents’ Chelsea parties and the affair between Jane and Tony – the letters in which Tony somewhat self-indulgently confesses to Karl, and Karl then responds, read like something out of early Margaret Drabble – it struck me that middle-class marriages are (paradoxically, perhaps) less flexible now than then: people seem to be so much more absolutist these days in matters of the heart. Does he agree? Yes, and no.

“We always have this idea that the generations are different, and I don’t know if that’s true.” On the other hand, people tend no longer to go along with the adage “least said, soonest mended” (his book is full of silences, quietnesses that to me seem to be rather sensible on the part of those involved). “Everything does become too hard and fast very quickly nowadays. On that, I agree absolutely.”

As to his response to the truth about his father (I’ve told him that his pragmatism and acceptance, at least as depicted in his book, make him seem sane to an almost unnerving degree), perhaps it helped that he wasn’t then too interested in inner feelings.

“I don’t know that sanity was involved,” he says. “My 16-year-old self would be shocked by my 55-year-old self writing so openly. It’s more that I only worked out how to be a human being at a later stage. I was rather an odd child, good at maths and able to get on better with adults than with other children. But then, one thing Karl and I had in common was that neither of us cared much what the outside world thought of what we were doing.” He looks very like White, and this amuses him on the rare occasions that he thinks about it. But mostly, he has no interest in trying to work out which bits of him come from where. Why not? It’s almost with a kind of longing that he says it: “Because Karl Miller was always my father.”

• Fathers by Sam Miller is published by Jonathan Cape (£14.99). To order a copy for £12.74 go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99