When Madonna Louise Ciccone arrived in New York from Michigan aged 19 in 1978, she had a small suitcase, a winter coat and $35 in her pocket. In an interview with the broadcaster Howard Stern years later, she would admit that she was frightened: “The massive scale of New York took my breath away … I was poised for survival … But I was also scared shitless and freaked out by the smell of piss and vomit.”

In the ensuing years, Madonna would experience grinding poverty, living in a succession of cockroach-infested apartments where addicts populated the hallways, and often going without food. Yet Whitley Setrakian, Madonna’s roommate at the University of Michigan, where both had studied dance, says: “I’ve never seen her so happy and so sure.” She recalls Madonna telling her: “Every day I’m here in this city I can’t stop thinking about all the days I haven’t been here. I feel like I’m running a race and everyone’s had a head start.”

This self-assurance, ferocious drive and refusal to let circumstances get the better of her echoes through Mary Gabriel’s doorstopper biography. The world is not short of books documenting the life of the queen of pop, though few have attempted it on such a mammoth scale. Coming in at more than 800 pages, the book leaves no stone unturned, and no song, music video, film, TV appearance, friendship or romantic liaison unanalysed, in its quest to understand the woman behind the global icon. This is Our Lady made flesh.

Gabriel, a former Reuters journalist, is the author of 2018’s Ninth Street Women, about five overlooked female artists, including Lee Krasner and Elaine de Kooning, which provides a richly detailed account of the creativity of postwar Manhattan. In A Rebel Life, she delivers a similarly evocative portrait of New York in the late 1970s and early 80s, a grimy, neglected city of chancers and ne’er-do-wells, and a hotbed of art and hedonism. The book is peppered with contextual digressions into the history and politics of the era, from the anti-Vietnam protests and Stonewall to the pill, the Equal Rights Amendment, Roe v Wade and the Aids crisis.

Those who insist Madonna was purely interested in fame may adjust that view when presented with her campaign to educate the world about HIV and Aids at a time when her contemporaries, not to mention politicians, were either silent or loudly disgusted. A full year before the US government began distributing leaflets about Aids, Madonna – who lost scores of friends to the disease, including the artist Keith Haring – was putting pamphlets about safe sex in brochures at her live shows, risking her reputation and career in the process.

Other perceptions of Madonna – that she “stole” her ideas from queer and black culture, that she needs a man in the studio to make her sound good – are deftly dispatched. On her supposed co-opting of the ballroom scene with her 1990 single Vogue, the author notes that two of the scene’s leading lights, Luis Camacho and José Gutierez, appeared in the song’s video and went on to tour with Madonna.

Gabriel’s writing is unfussy and direct – the approach of a cultural historian rather than fan – while delighting in details such as Madonna getting fired from her job at New York’s Russian Tea Room for wearing fishnets, or sticking her tongue in Al Pacino’s ear after meeting him as a bolshy 20-year-old.

The figure who emerges is inevitably contradictory: serious, funny, generous, inconsiderate, hard-working, restless, blinkered, self-possessed, stubborn. This is a story not just of hit records, reinvention and headline-grabbing controversy, but about how a young working-class woman, traumatised by the early death of her ultra-religious mother, rose to become a cultural colossus and thorn in the side of the conservative right and the Catholic church. Underpinning the narrative are the repressive patriarchal systems that Madonna relentlessly challenged and her unwavering belief in the right to self-expression.

Along with music, that self-expression also found an outlet in film, with decidedly mixed results. There can be no better illustration of Madonna’s doggedness than her acting career, where the howling duds (Who’s That Girl?, Body of Evidence, Swept Away, The Next Best Thing) vastly outnumber the critical and commercial smashes (Desperately Seeking Susan, Evita). Yet on she went, determined to prove her critics wrong.

It’s a mark of Gabriel’s skill that she has managed to wrestle this complex, sprawling, eventful life into a book that rarely flags and conveys its subject’s wider significance without tipping into hagiography. We come to understand Madonna the person as well as Madonna the concept: a woman who, for a generation, embodied female artistic, sexual and financial liberation.

“Just as the daughters of second-wave feminism, who had been taught that they were equal to their brothers, started learning that was not the case, they discovered Madonna,” writes Gabriel. “Her look gave them a rebellious way to express their simmering dissatisfaction and emerging defiance … The message wasn’t the 1970s ‘I Am Woman’ but a 1980s ‘I am free.’ But what did freedom look like? It looked like Madonna.”

• Madonna: A Rebel Life by Mary Gabriel is published by Coronet (£35). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply