In 1990 a flyer circulated in San Francisco inviting people to attend a unique happening: “The Zone Trip is an extended event that takes us outside of our local area of time and place. On this particular expedition, we shall travel to a vast, desolate, white expanse stretching onward to the horizon in all directions.”

Burning Man, the weeklong counterculture festival and arts jamboree, has continued to take place each August, with some 70,000 people converging on the dry bed of an ancient lake in Nevada’s Black Rock Desert. The event is not what it was: the drive-by shooting range is gone, tickets cost hundreds of dollars, and there is a real danger of running into Elon Musk. But its founding principles are still upheld. These include “radical inclusion”, “radical self-reliance” and “radical self-expression”.

Fifty years on from the first publication of Desert Solitaire, Edward Abbey’s classic account of his time as a ranger in Utah’s Arches national park, the desert still represents freedoms unavailable elsewhere. Abbey could evoke the raw, alarming beauty of arid landscapes as effectively as any writer, but he was also capable of startling cruelty and obtuseness, particularly towards immigrants. In a 1977 essay titled The Great American Desert he at once extols the desert’s appeal and discourages those who would follow him there. “Survival hint #1: Stay out of there. Don’t go. Stay home and read a good book, this one for example.” As well as the bloodsucking kissing bug and a half-dozen species of rattlesnake, there are black widows, Gila monsters, the deadly poisonous coral snakes and the giant, hairy desert scorpions. “Something about the desert,” he adds, “inclines all living things to harshness and acerbity.”

Chief among Abbey’s literary forebears was art historian John C Van Dyke, who in The Desert (1901) wrote a treatise that transformed America’s vision of its arid south-west. It was an act of aesthetic rehabilitation of a realm that had been deemed not only valueless but inimical to the national spirit of Manifest Destiny. For the first time, the American desert was described in terms that not merely eulogised its beauties but proposed it as an image of transcendence. A longing for solitude extended to Van Dyke’s writing: “Heaven knows the literature of humanity is large enough without dragging it into such sublime isolations as the desert.” Accordingly we meet hardly a single living, breathing human in his book.

If Abbey inherited one vice from Van Dyke it was that “harshness and acerbity” he observed of the land itself, a conviction that the desert – not to say the planet – would be perfect but for its one defining taint: humankind. TE Lawrence (“Lawrence of Arabia”), too, writing to Ernest Thurtle in 1929, insisted that “what is wanted is a new master species – birth control for us, to end the human race in 50 years – and then a clear field for some cleaner mammal”. Tongue-in-cheek, perhaps, but this was the Lawrence who, crossing the Sinai in 1917, saw himself mirrored in his Bedouin companions: “His sterile experience robbed him of compassion and perverted his human kindness to the image of the waste in which he hid … he found luxury in abnegation, renunciation, self-restraint.”

Another British traveller in the Middle East, Wilfred Thesiger, recalling his marathon journeys in the Arabian peninsula’s Empty Quarter in the 1940s, wrote: “Sex has been of no great consequence to me, and the celibacy of desert life left me untroubled.” For Thesiger and Lawrence, as for the Desert Fathers who founded Christian monasticism in third-century Egypt, the desert promised sanctuary not just from society but from the importunate, unclean body.

The confusion of flight with quest is a theme of many contemporary desert texts. Taking for granted canonical dryland novels such as Cormac McCarthy’s Blood Meridian (1985) and Michael Ondaatje’s The English Patient (1992), a modern desert library might include Jim Crace’s hallucinogenic novel of Christ’s years in the wilderness, Quarantine (1997), French-Egyptian Jewish poet Edmond Jabès’s exquisite desert koans written during the 1980s from his exile in Paris and Sven Lindqvist’s anti-colonial Saharan odyssey, Desert Divers (2002). In Kim Mahood’s 2016 “memoir-map”, Position Doubtful, the artist, returning to the Australian interior of her childhood, describes a “sense of the desert as a presence in my body, as if in my trajectory across it I have absorbed it”.

The modern literature of America’s south-west alerts us to the land’s contested nature. The desert in Charles Bowden’s astonishing apocalyptic non-fiction trilogy Inferno, Exodus/Éxodo and Trinity (2006–9) is a boneyard scattered with the remains of slaughtered narcos and lost migrants, and trembling under the legacy of nuclear testing. Rebecca Solnit’s 1994 Savage Dreams reveals the “hidden wars of the American West”, including the 953 nuclear tests at the Nevada Test Site that “turned the desert itself into a scapegoat for the political conflicts of a nation”. Published this year, former border patrol agent Francisco Cantú’s lacerating memoir The Line Becomes a River is an artful reminder that America’s border deserts continue to be weaponised against undocumented migrants attempting to enter the US from Mexico.

The title of Hari Kunzru’s 2011 novel Gods Without Men references Balzac’s description of the desert as a place “where God is and man is not”. A hyperactive, polyphonic, time-twisting excursion into arid California, it opens with a modern fable. Sick of his life, the trickster Coyote god of Native American myth decides to “go out into the desert and cook”. What he cooks is the drug methamphetamine – crystal meth. In an RV converted into a mobile lab, assisted by “Cottontail Rabbit” and “Gila Monster”, Coyote eventually – after twice accidentally killing himself – synthesises “a hundred grams of pure crystal”.

The story recalls a 2009 episode of the TV series Breaking Bad, “4 Days Out”, in which Walter White, chemistry teacher turned meth-cooker, believing he is dying from lung cancer, persuades his young associate, Jesse Pinkman, to help him prepare a consignment big enough to provide for his family after his death. With Blue Mink’s “Good Morning Freedom” blasting from the RV’s stereo, they drive out of Albuquerque until there is nothing but scorched grassland stretching to every horizon.

This is the classic American desert, a frontier defined less by an absence of water than of oversight – what the great scrutineer of the west Mary Austin called a “country of lost borders”. (It is also a country of global nightmares: 120 miles south of Albuquerque is the Jornado del Muerto desert, site of the world’s first nuclear explosion, in 1945.) Inevitably, the RV’s battery dies and they run out of water. When Walter vanishes, Jesse finds him staring into the desert, a stigmata of lung-blood hawked into one palm.

In ‘The Great American Desert’, Abbey asks: ‘Why the desert, given a world of such splendour and variety?’ Then he tells a story. Hiking in Arizona, he climbed the wall of a canyon and found laid out on the rim an “arrow sign, three feet long, formed of stones”. It pointed in the direction of “sand and desert rock under the same old wide and aching sky”. Abbey reckoned it had been there for at least a century. Searching for whatever the arrow was pointing to – “an ancient Indian ruin, a significant cairn, perhaps an abandoned mine” – he finally understood. The arrow pointed to “nothing at all. Nothing but the desert. Nothing but the silent world.”

Anyone who has been lost in the desert might wonder if the arrow was not simply a way-marker, but the Desert Fathers (there were some Mothers, too) were familiar with Abbey’s ‘nothing at all’. For them it was paneremos, the ‘uttermost desert’, while French travellers in the Sahara in the eighteenth century knew it as le désert absolu. ‘The desert in its absence of the things of this world manifests the presence of the next,’ as Solnit puts it in Savage Dreams.

The image of Walter and Jesse’s immobilised RV seething in the American nothing evokes not only the pair’s isolation, but the landscape’s crushing indifference. As Abbey knew, “the desert doesn’t give a shit”. The French cultural theorist Jean Baudrillard, travelling through the same landscape in the 1980s, observed that “the silence of the desert is a visual thing, too. A product of the gaze that stares out and finds nothing to reflect it.” In Breaking Bad, the stillness of the desert and its silence are heightened by the fitful buzzing of a fly or a raven’s caw, and by a soundtrack that resembles something being heated until it sings – aural mainstays of desert movies from Nicholas Ray’s Bitter Victory (1957) to Wim Wenders’s Paris, Texas (1984), Mahamat-Saleh Haroun’s Daratt (2006) and this year’s Lean on Pete. For Baudrillard, the American desert was inextricable from the cinematic desert. “It is useless,” he wrote in America (1988), “to seek to strip the desert of its cinematic aspects in order to restore its original essence; those features are thoroughly superimposed upon it.” Wobbling like a mutating cell, the sun hanging above the dehydrated pair is an image straight out of David Lean’s 1962 biopic Lawrence of Arabia, and I wonder if it’s just coincidence that Walter wears his own pitiful version of Lawrence’s Bedouin headdress: a soiled towel wound with a bungee cord.

Jesse might by now be regretting skipping a visit to the Georgia O’Keeffe Museum in Santa Fe, suggested by his girlfriend back in Albuquerque. O’Keeffe first visited Taos, New Mexico, in 1929, buying a house there in 1940 and finally leaving New York and moving to the state permanently in 1949. To walk in a desert – assuming you have water and are not lost – is to be a kind of beachcomber: the wonder of each object’s “thingness”, its particularity and significance, are amplified by the environment’s very sparsity. O’Keeffe’s paintings often focus on these singular findings: a crucifix, a tree, a flower. Her famous skulls, she insisted, were not memento mori: “The bones seem to cut sharply to the centre of something that is very keenly alive on the desert, even tho’ it is empty and untouchable.”

This arid landscape, for O’Keeffe, knew “no kindness with all its beauty”; but as a marginal realm the desert has always been confidant and consolation to those at society’s margins. Between 1992 and 2007, photographer Laura Aguilar, who died in April, took several series of nude self-portraits in the deserts of New Mexico, Texas and California. Aguilar – gay, Mexican-American and with long-term ill health – produced a body of work that, like O’Keeffe’s, asserts an unanswerable claim on cowboy territory. In her images she is folded and slumped over rocks, or crouched in what might be one of O’Keeffe’s dry washes, her face concealed, her monumental body at once echoing the desert’s geology and strikingly organic – like the land itself, seemingly dead and alive at the same time. Setting the hairless human form against the sparsely vegetated land, Aguilar’s work is not Lawrence’s quest for “abnegation, renunciation, self-restraint” but rather a celebration of the rapport that can exist between body and topography.

If the desert can function as a symbol of transcendence, it is also a symbol of death: silent and apparently inert, topography picked away to its geological bones. As long ago as 1901, Van Dyke looked heavenwards and asked: “Have we not proof in our own moon that worlds die? Perhaps it died through thousands of years with the slow evaporation of moisture and the slow growth of the desert.”

By 2020, 60 million people are expected to have migrated from sub-Saharan Africa alone due to the degradation of fertile land by overgrazing, cutting down trees, mining, unregulated irrigation and climate change. Desertification, as these processes are known collectively, is described by the UN as “among the greatest environmental challenges of our time”.

If Van Dyke’s science was flawed (the moon, unlike Mars, has always been dry), his questions might unsettle even the most dreamy solipsist, gazing out at the desert in 2018. “Is then this great expanse of sand and rock the beginning of the end?” he asks. “Is that the way our globe shall perish?” •

• The Immeasurable World: Journeys in Desert Places by William Atkins (Faber & Faber, £20). To order a copy for £17, go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.