Not again,” said the great American architect Louis Kahn, when his lover Harriet Pattison told him she was pregnant. He’d had another lover, Anne Tyng, who had had a daughter with him eight years before. It is one of the lines in Pattison’s book that stings. Another comes when she finds that his wife Esther, who was relatively prosperous, was helping to support the precarious finances of Kahn’s studio. It meant that Pattison’s hope that he would leave his wife for her was pushed into a remote and, as it turned out, never achieved future.

Yet Our Days Are Like full Years is not a bitter or angry book. It is a memoir of their times together, moving and heroic (on her part) as well as troubling, built around the letters he sent to her, which she has kept in a Chinese cinnabar box ever since he died in 1974. Their relationship lasted 15 years – “a small portion of time in most lives, and yet measureless in intensity and effect in ours”. Like much of her life, she says, the letters “were unexamined, put off for a later time. But it is a later time now, for I am past 90.”

This relationship owed much to her forbearance. Kahn, she says, “could not be pinned down, but was always on the move, trying to engage with the world and make it better through his art”.

“I am in a dream world which no one will share,” was how he explained himself, “yet I know it is human and true.” He lived for his work. While Pattison, as a single mother, was struggling to build her career as a landscape architect, he was embarking on marathon journeys to visit his projects and give talks: Karachi-Tehran-Moscow-Warsaw-London. He would send her postcards and notes, often signed off in a rush.

At one point Kahn was designing the National Assembly in Dhaka, East Pakistan, later Bangladesh, which became one of his most significant projects. At the same time he was working on a (never realised) presidential house in Islamabad, West Pakistan, and a major building in Ahmedabad, India. As these countries slid towards the 1971 war that would make Bangladesh independent, Kahn attempted ingenious ways to travel from one to the other without his clients’ knowledge. It’s easy to see a parallel with his nocturnal journeys round Philadelphia, visiting the households of his wife, lover and former lover.

Pattison is generous. She attributes his restlessness to his childhood, as an impoverished immigrant from Estonia, forever moving on from one tenement to another. As if in compensation, his architecture was strikingly permanent and rooted, inspired by the massive masonry of Egypt, Rome and medieval cathedrals, but it also had the quality of ruins, with big unglazed openings, as if not fully inhabited.

The question was whether his creativity, together with flashes of humanity and warmth, were sufficient compensation for the pain that Kahn plainly caused. These were the themes of My Architect, a 2003 film made by their son Nathaniel. Now, Pattison tells her story, and she’s clear that their time together was worth the frustrations.

The book ends with the 2012 opening of the Four Freedoms Park on Roosevelt Island in New York, a long-delayed memorial to FDR on which Pattison and Kahn collaborated towards the end of his life. “I am overwhelmed by what has been achieved,” she writes. “My great longing was to live a life in art, and with Lou’s help I had found a way to do it. This work allowed me to transcend myself and be part of something bigger and more lasting.”

• Our Days Are Like Full Years: A Memoir With Letters From Louis Kahn by Harriet Pattison is published by Yale University Press (£35). To order a copy go to guardianbookshop.com. Delivery charges may apply