Spare a thought for the sleepless nights of Mr Edward Shrimpton, a self-made builder, who by 1960 had bought himself a 200-acre farm in Buckinghamshire, and sent his two daughters to the best local convent school. First, his eldest, Jean, 18, falls for a married man, an East Ender, who has been taking “glamour” pictures of her. And if that weren’t enough, his younger daughter, Chrissie, is knocking around with a student, whose only prospect of gainful employment seems to be with a music group who are yet to release a record. Faced with the prospect of David Bailey in his hay barn and Mick Jagger in his back bedroom, it seems that Edward Shrimpton could not decide who to aim his shotgun at first.

Sixty years on, Jean Shrimpton and Bailey can still conjure the thrill of those early liaisons, in which at least one free-spirited image of the decade that followed was created. They never did an interview together in the years that they made each other famous, as far as they recall, so when they sit down here, at the heart of Bailey’s predictably candid memoir, it’s like they are piecing together little fragments of a shared fantasy: the landmark Vogue shoot in New York, taking studio fashion on to the street, him bringing flowers to her, and carrying them without embarrassment, their first sex, on Littleton Common (“ah yes, I remember it well”). “It took me three months to get my leg across,” Bailey recalls, to emphasise his gallantry.

There is much that you might feel you know already in this book, but what lifts it above the familiar are the recorded segments in which we hear Bailey test his memory against some of the people he has been closest to – not only Shrimpton, but also his ex-wives, Catherine Deneuve and Marie Helvin, current wife, Catherine Dyer, and many more friends and lovers. In this way James Fox, Bailey’s ghostwriter, makes what could be an exercise in self-mythology something far more raw and surprising (in this sense, the book makes a worthy successor to Fox’s last efforts as a “ghost” for Keith Richards’s Life). At times he gives full rein to Bailey’s unreconstructed cockney persona, but eavesdropping on him in the company of old friends and old lovers, we also hear him confront and examine some more complex drivers of his priapic creativity.

A lot of this goes back to the photographer’s childhood. Fox probes and presses him about his relationship with his parents, his angry mother, and his wayward father – he was the accident that enforced their horribly unhappy marriage – and lets that wartime rationing of affection explain some of Bailey’s legendary restlessness. Bailey dates his eye for female beauty back to the afternoon in 1948 when he went with his mother and his aunt on their annual trip to the West End, and to Selfridges to try on the dresses they could never afford, before they returned home to run up versions of them on their sewing machines. On one such occasion his mother tried on a Dior-style dress and twirled in front of the department store window, backlit – and nine-year-old Bailey took his first photograph in his head.

He held on to that image, it seems, even as he was being called stupid every day at school (he left at 14 and was diagnosed with dyslexia aged 30) and through jobs that included a spell as a debt collector in the East End in his teens. He was, according to his oldest friend, Danny O’Connor, never without “the Bailey stare” apparently seeing things in people – women – that they hardly saw themselves. The stare got him into lots of trouble, taking beatings from local thugs, “the Barking boys”, whose girlfriends he eyed up, but it was also his almost unfailing attraction. He never doubted that gift. “My mother used to say, ‘You’ll end up like all of us, driving the 101 bus,’” he recalls. “I thought, ‘I fucking won’t.’”

Much of this story, and much of the behaviour it describes, reads like a foreign country in our more censorious times. As Bailey’s longtime assistant John Swannell puts it: “Whatever trip he was on, he went for the model and 99% of the time he was bonking them.” There is, from the many former lovers who appear in these pages, no suggestion that any of that attention was unwanted. Fox’s subtle telling lets you see how this looked through Bailey’s eyes, and occasionally allows you to see the edges of that viewfinder.

• Look Again by David Bailey is published by Macmillan (£20). To order a copy for £17.40 go to guardianbookshop.com. Delivery charges may apply

• This article was amended on 2 November 2020 to remove an incorrect reference to Mick Jagger having been an art student.