Laura Ingalls Wilder’s Little House in the Big Woods was first published in 1932, when its author was 65; it offers a sanitised tale of her childhood near Pepin, Wisconsin, just after the end of the US civil war. Within a few years of her birth, the Ingalls family piled their few possessions into a covered wagon and started the trip into “Indian Territory”, to join the settlers pushing west in order to make manifest the destiny that America was determined to invent. Six more books followed, detailing the family’s experiences on the frontier, creating an idealised, nostalgic account of Laura’s peripatetic early years, along with one book describing her husband Almanzo Wilder’s childhood on a farm in New York. Wilder died in 1957; her daughter, Rose Wilder Lane, published Wilder’s unfinished final novel, The First Four Years, in 1971, and three years later the immensely popular Little House on the Prairie series debuted on US TV.

But before all that was Pioneer Girl, a memoir that Wilder wrote for her daughter in 1930, and which has just been published for the first time. The 1929 crash had left her family in financial straits, and Wilder had been publishing a small domestic column in local magazines for some time. With a dawning sense that her own experiences exemplified the US story of westward expansion, she set down her memories, from the age of three through to her marriage to Almanzo Wilder at 18, in hopes that her tale might find a publisher. When this proved impossible, she and Lane, a successful writer and experienced editor, began discussing the possibilities for adapting the story into children’s books that would follow the progress of young Laura from childhood to adulthood. Gradually, Wilder’s artistic instincts and skill improved, and she took over more of the writing and editing, but Lane remained an important interlocutor for her mother’s developing sense of plot and character. Eventually, Wilder would explain that her fictionalised chronicles were not “a history, but a true story founded on historical fact”.

Pioneer Girl offers more history and less fiction: it is presented as Wilder first wrote it, complete with asides to her daughter, no section breaks, and spelling mistakes (an ironic aspect for readers who remember the novels’ emphasis on Laura’s spelling bee triumphs). Carefully, not to say exhaustively, annotated with research into the places they settled and the people they encountered, Pioneer Girl provides a fascinating counterpoint to Wilder’s sterilised chronicle of sunny life on the open prairie. The reality, unsurprisingly, was rather more vicious.

One of the most memorable incidents in the novels comes in the final instalment, These Happy Golden Years, when 15-year-old Laura, boarding with an unhappy, squabbling family named the Brewsters, awakes one night to the sight of Mrs Brewster in a trailing nightgown, hair streaming behind her, wielding a butcher knife over her husband and demanding that he take her back east. Laura is terrified, but who wouldn’t hate homesteading? They were squeezed together in a one-room shanty on the howling prairie with a squalling baby, braving blizzards and temperatures that routinely froze the thermometer at -40C, surviving on salt pork and fried bread. You can understand the impulse to wield a knife.

What seems remarkable is not Mrs Brewster’s rage, in other words, but the cheerful acquiescence of the Ingalls women to such conditions, and their ability to create a series of safe, “cosy” homes that enable them to survive the hostile environment. Viewed from one perspective, Charles Ingalls’s wanderlust seems almost pathological, and certainly selfish. Pioneer Girl reveals that the incident with Mrs Brewster was just one in a succession of encounters with serious domestic violence. For most of Laura’s childhood, she lived in close proximity to drunks, rapists, horse thieves, adulterers and more than one murderer, including perhaps a brush with a notorious family of serial killers.

Nor was the Ingalls family’s progress a simple westward expansion, as the novels more or less report it: Wilder deliberately simplified their back-and-forth journeys across the midwest in order to create the impression of westward progress, an image in keeping with her theme of nation-building. In fact, the Ingallses retreated east more than once, while the self-styled “pioneers” were land-grabbing as fast as they could: manifest destiny was a giant get-rich-quick scheme. The family emerges as far more opportunistic, even on occasion unscrupulous, than the whitewashed novels would have us believe. Charles Ingalls knew that he was in Indian Territory illegally, while his wife’s brother Tom went to the Badlands on an ill-fated and illegal search for gold. The family snuck away from debts on at least one occasion, making their escape in the middle of the night. The fundamental drama in the novels comes not from conflicts within the family, but with the external forces of government, Indians and nature. Of the three, the last is the one we are most apt to sympathise with today; the attitude of the settlers to the Indians makes for uncomfortable reading (“Treaties or no treaties, the land belongs to folks that’ll farm it. That’s only common sense and justice,” one neighbour declares). It is an attitude that the Ingallses and their relatives clearly shared; many of their actions were in flagrant violation of treaties.

The most extensive difference between the two accounts is Wilder’s decision to excise an entire interlude in Iowa from the novels. The family’s retreat to the east undermined her triumphalist tale of westward progress, but their time in Iowa also featured some of the family’s grimmest experiences. They lived in a hotel adjacent to a saloon, which is hard to imagine the fictional Ma Ingalls permitting; the decision was a mark of their “financial desperation”, as the editor of Pioneer Girl notes. There were bullet holes in a wall, made by a drunken man shooting at his wife; another dragged his wife around by her long hair, carrying a lamp that was pouring kerosene; Charles Ingalls intervened to keep them all from being burned to death. A man named Hairpin, “who had been lying there drunk for several days, came to and took another drink to sober up”. With the whiskey still in his mouth, he lit a cigar and inhaled the flames, which killed him “almost at once”. In Iowa “Christmas was disappointing”, for “Ma was always tired; Pa was always busy”. This seems much more realistic than the always inspiring Christmas tales in the books, in which kind neighbours consistently come to their rescue, or the family pulls together and makes merry for each other. (Beloved Mr Edwards, the kindly neighbour who memorably saves one Christmas, is nowhere to be seen.) A man who’s been drinking gives a temperance lecture with a bottle of whiskey in his pocket.

There is much more illness than in the books: more than one bout of scarlet fever, whooping cough, measles, mites and the meningitis that blinds Mary, which Wilder transformed to scarlet fever in the novels (perhaps, the editor speculates, to link her tale to Little Women). Lane wanted her mother to excise Mary’s blindness altogether, but Wilder was adamant, insisting “a touch of tragedy makes the story truer to life”. She did, however, erase the entire existence of her little brother, Freddy, who died as an infant while they lived in Iowa. Children are lost in blizzards, freeze to death or lose limbs. Unsurprisingly, the girls are far naughtier than in the novels: even saintly Mary is spanked for yelling as a child, while Laura bites her cousin till his thumb bleeds for washing her face in snow. In the novels, Laura doesn’t leave home for work until she is an adolescent, but in reality she was sent as a child to stay with strangers as a babysitter and paid companion. Once a drunk man came into her bedroom in the middle of the night and told her to lie still. She threatened to scream, and the next day was taken home. While such incidents may be more realistic than the sentimentalised novels, Wilder’s writing in Pioneer Girl is flat and amateurish: she had not yet learned to slow down and tell her story, creating sufficient time and space for readers to get to know characters, to identify with their travails, to build drama and sympathy for them.

Pioneer Girl’s annotations and footnotes make it unwieldy in more senses than one. The notes far outweigh the narrative, running along both sides of an awkwardly wide page and on many pages displacing the primary narrative altogether. They are packed with information, sometimes excessively so (we probably don’t need to be told what braille is, or what “idiot” meant in the 19th century). All of the fascinating historical research into what happened to the people Wilder encountered, such as Cap Garland (killed at 26 when a threshing machine exploded) or Nellie Oleson (an amalgamation of at least two girls Ingalls knew) is buried in the notes, which makes for an extremely disjointed reading experience.

In the end, the changes that Wilder made to improve her story remained consistent with the truth of her own experience. “Even though these books must be made fit for children to read,” she told her daughter, “they must also be true to history … I have given you a true picture of the times and the place and people. Please don’t blur it.” The publication of Pioneer Girl has done more to keep the historical picture distinct, dispelling some of the mists and myths of legend and showing us the dark realities of US pioneer life.

• Sarah Churchwell is the author of Careless People: Murder, Mayhem and the Invention of The Great Gatsby.

• To order Pioneer Girl go to pioneergirlproject.org