James Hamilton-Paterson is among the most reclusive and mysterious of British literary exiles. A loner by temperament, he belongs to no metropolitan coterie or salon, and for the past 25 years has ploughed his own furrow. His work resists definition. Travel? Autobiography? Fiction? "I suppose it has elements of all three," Hamilton-Paterson says, adding: "Publishers find it hard to pigeonhole me." His books have won him a cult readership, nevertheless. In addition to five non-fiction works, he has published two volumes of poetry, five novels, three children's books, two short story collections. He also writes a fortnightly column on marine and scientific matters for a Swiss magazine. Now 62, Hamilton-Paterson left his native England over a quarter of a century ago, and divides his time between Tuscany and the Philippines. He describes himself (with characteristic self-deprecation) as a "rat-poor literary drifter" and "professional absentee".

Though he remains elusive, Hamilton-Paterson has long hovered on the edge of the literary mainstream. He attracted a wider audience in 1989 with publication of Gerontius, his spellbinding reconstruction of a journey made by the composer Sir Edward Elgar along the Amazon in 1923. Regarded by admirers as being among the best British novels of the 1980s, its glowing and luxuriant delight in words and dreamlike landscapes deeply impressed the novelist Michael Ondaatje. "Gerontius had this wonderful mongrel quality - part travelogue, part Conradian shipboard drama - and straddled genres in a very exciting way," Ondaatje says. "No other modern British novel I'd ever read was anything like it." The director Jonathan Kent planned to make a film of Gerontius with Paul Schofield as the vulnerable, petulant Elgar; nothing has come of the idea yet.

JG Ballard is another who has long admired Hamilton-Paterson. "I love his elegant and intensely evocative style: strangeness lifts off his pages like a rare perfume." His reluctance to publicise himself, along with his scant regard for literary fashion and mass market tastes, has made him all the more alluring, Ballard believes. "He's one of the very few writers alive today who retains his mystery." Barry Humphries, the Surrealist comic, is similarly entranced. "Hamilton-Paterson hasn't been interviewed by Melvyn Bragg and he doesn't work the literary circuit. He appears to be an anchorite. Yet I'm bowled over by the sheer imaginative brilliance of the man."

Though he has travelled widely, Hamilton-Paterson is best known as a commentator on the Philippines, where he has lived on and off since 1979. His grand-guignol thriller, Ghosts of Manila (1994), portrayed the Philippine capital in all its ravishing decay and street violence. Alex Garland and other literary hipsters with a taste for South East Asian crime and politics have said it influenced them. Yet, says the journalist Lynn Barber, Hamilton-Paterson is still seriously underrated in Britain. "I've known him for a good 10 years and literary fame could have been his for the asking. Still, I suspect a part of James rejoices in the fact that nobody quite knows him through and through."

For much of his writing career, Hamilton-Paterson has chosen to live at a remove from his publishers. His solitariness has probably not helped his sales, and books including Seven-Tenths have fallen out of print. Barry Humphries thinks his books are so "strange" and "undefinable" that inevitably they constitute "caviar" and remain a minority enjoyment. When not subsisting in a bamboo hut in the Philippines, Hamilton-Paterson's home until recently was in such a remote corner of Tuscany that it could be reached only by jeep. Today he lives in less spartan Tuscan quarters, near Pisa airport: "The hills behind me are awash with British house queens who do up their farmhouses and set up watercolour easels. I'm fond of Italy but I can't stand all that sentimental, Toujours Tuscany guff."

Hamilton-Paterson has drawn on his experience of "Tuscminster" for his latest novel, Cooking with Fernet Branca, a deliciously nasty farce set not a million miles from Pisa, which represents the author's first bid for mainstream recognition. "I wasn't aware that the book was so especially funny until people started telling me how uproarious it was," Hamilton-Paterson says. The book is partly a satire on the "John Mortimer brigade", as he calls the British holidaymakers who descend on this part of Italy in summer.

At first glance, the novel could not be more different from Hamilton-Paterson's other books, with their beautifully transparent prose and enchanted explorations of other worlds. Neil Belton, the author's editor for more than 10 years, says you can nevertheless hear the thrust and parry of Hamilton-Paterson's conversational manner in the book's witticisms and black comedy: "For an avowed recluse, James is a brilliant talker, at times flamboyant, and in his new novel he gives full rein to that side of his personality."

Hamilton-Paterson's background is upper middle class. He speaks in clipped, patrician tones, occasionally with a bluff jokiness, and is a surprisingly genial host. His father was born in 1915 in China of medical missionaries, and sent back to England at a young age to be educated at Eltham College, south London, where he was a contemporary of future novelist Mervyn Peake. His mother was a debutante whose parents lived in a house off Sloane Square. She had an "anti-establishment" streak, though, and rather unusually for those pre-feminist days chose to pursue a career in medicine. It was she who taught Hamilton-Paterson's father anaesthetics at University College Hospital, where he was a pre-war medical student.

Their son James was born in 1941 in the north London suburb of Stanmore. He was no sooner born, however, than his father was called up and sent to India as a major in the Royal Army Medical Corps. Hamilton-Paterson did not see him again until he came home a demobbed stranger at the war's end. Hamilton-Paterson was four. "I regarded him as an interloper who took charge of - and monopolised - my mother's affections," he says, "and I suspected I was not the son he would have liked." There were other frustrations too. "I resented the fact that my father seemed to oblige me to have these awful tantrums. He was always so overworked and crusty and irritable that we ended up having slanging matches. But I didn't want to shout - shouting wasn't me at all." Try as he might, Hamilton-Paterson could not come to an easy intimacy with his father, and the fact rankles with him still. "Boys, especially small boys, need their fathers. My father behaved like a gent, but a very distant gent all the same." Hamilton-Paterson's younger sister, Jane, to whom he remains close, was born in 1943.

In 1949, at the age of eight, he was packed off to a South Coast preparatory school near Worthing. There was "ritualised bullying" and Hamilton-Paterson felt cruelly wrenched from Footscray, in genteel Kent, where his family now lived. "The school's elderly head was dying of liver cancer and basically he'd lost control. A good deal of quite violent sexual stuff went on - perpetrated by older boys." Hamilton-Paterson implored his parents to remove him. His mother was sympathetic, his father less so. The bullying continued and eventually he was transferred to Bickley Hall, a school nearer his home in Kent.

He became captain of the Bickley Hall rifle team and thinks he might have made "quite a good sniper". In the summer holidays he manufactured explosives out of fertiliser, and became a practised bomb-maker. His father pointedly turned a blind eye to the high-risk detonations. "I wonder now whether at some unconscious level he wasn't hoping that I might blow myself up," Hamilton-Paterson hazards. Yet father and son had some interests in common - both were entranced by ships and water. One summer's day in 1950 Hamilton-Paterson was taken down the Thames by his father on a London water bus. "It was the first time I'd ever been on the river and I was smitten by the coal-gas smells, the sour tidal flats and warehouses and winches," he recalls. Over the coming months, fortified by his reading of Robert Louis Stevenson, Hamilton-Paterson's head filled with aquatic marvels and dreams of faraway places. "The lure of exotica - of spices and foreignness - became very, very powerful for me." Forty years later, in 1992, he published his exquisite, far-ranging meditation upon the sea and its meanings, Seven-Tenths, praised by William Golding for its intensely poetic "abstractions". Among other things, the book is a lament for the death of sea travel, and his admirers claim that it contains some of the finest writing about the sea and man's umbilical link with the deep since Joseph Conrad.

In 1955, having passed his common entrance, Hamilton-Paterson went on to King's School, Canterbury, where he discovered he had an ear for music. "I'd wanted to be a classical composer," he says, "but gave up when I realised I didn't have the requisite gifts." He improvised for hours on a school piano and thrilled to Tudor church music. Music was to play an important part in Hamilton-Paterson's books. A collection of short stories is called The Music, and Gerontius inevitably is shot through with reflections on music and composition.

With his facility for music came an extreme sensitivity to the musicality of words. "I spent my entire schooldays trying to conceal this awful weakness of mine, but whenever I was asked to read out loud an especially beautiful or euphonious poem, I started to cry." Tennyson's "Morte d'Arthur" had him weeping in class. "And, boy, did one get bullied and mocked for weeping at Tennyson." The poet's Victorian melancholy would influence his mature prose, with its sorrowful, incantatory tones and hard-won grace. Meanwhile, Hamilton-Paterson used his talent for verbal wit and sarcasm to deter further bullying. "I had a reputation for giving boys cruel nicknames or saying things that smarted. You can get a lot of mileage out of having a sense of humour."

He did not shine academically at King's School, and failed his maths O-level seven times. On leaving Canterbury he was unsure what career to pursue. National Service conscription was a frighteningly real possibility. An awareness of death had in fact come early to Hamilton-Paterson, when several boys of his acquaintance at Canterbury were killed or badly injured in Malaya or Kenya during Britain's last colonial wars. "There was a constant sense - we all had it - that sooner or later we'd be called up and might well die." He got out of doing National Service by just three months.

For a year after Canterbury, still uncertain what to do, Hamilton-Paterson taught at a prep school in Hertfordshire which was "barking mad, in an Evelyn Waugh sort of way". Teaching beckoned as a full-time career until, in 1961, he got a place to read modern languages at Exeter College, Oxford. During his first year at the university his father died, at 47, from stomach cancer; Hamilton-Paterson and his sister Jane felt relieved: "Thank God he's gone." Today Hamilton-Paterson comments: "I do feel slightly shocked by the vehemence of our reaction, and a part of me feels sad that I could never truly know my father." A powerful figure had gone from his life, and he became aware of his changed relationship with his mother as he and his sister now saw to her happiness and welfare. She died three years ago, at 92.

With time, Hamilton-Paterson has revised his opinion of his father, and in 1987, at the age of 46, he published his most personal book to date, Playing with Water. Ostensibly it describes a period of enforced solitude on a remote island in the Philippines, yet it brims with recollections of his father and of a bygone England of Wolseley cars and home-county proprieties. Typically, though, Hamilton-Paterson did little to promote his book.

Following his father's death, Hamilton-Paterson switched courses at Oxford to read English. His urbanely detached humour and gift for music were noticed by Jonathan Wordsworth, the young fellow in English at Exeter College who later tutored Martin Amis. (When Hamilton-Paterson last saw Amis, in the mid-1970s, the younger graduate asked his advice on whether or not to become a novelist. "Yeah, why not give it a whirl," he suggested.) In their early twenties, Jonathan Wordsworth and his wife Ann invited favoured undergraduates to lunch at their house in Warborough outside Oxford.

As Hamilton-Paterson remembers: "Warborough seemed to me an idyllic sort of intellectual-bohemian place. It was a bit squalid, sometimes, but no one seemed to mind if children ran round more or less unclothed." Leonard Cohen was on the turntable, and while other Oxford students wore tweeds and "calvary twills" (as Hamilton-Paterson calls them), the Warborough set wore jeans and Beatle boots. During this period, in preference to student dances, Hamilton-Paterson went to see Gerry and the Pacemakers and The Dave Clark Five at Oxford Town Hall. "I'd never met anyone else like him," Ann Wordsworth recalls of the student Hamilton-Paterson: "He was never bland. Neither was he eccentric, exactly, but he did stand out. Sometimes he played Bach and Mozart for us on the piano."

In 1964, on the eve of his graduation, he won the Oxford Newdigate prize for his poem "Disease", which memorialised his father's death a year earlier. At the awards ceremony he was seated between the poet Robert Graves ("who had shaving foam dried in his right ear") and Agatha Christie. Next year, 1965, he undertook postgraduate studies in education at King's College, London, with a view to more teaching. Afterwards he went to Libya to teach English at a school in an Arab neighbourhood of Tripoli. It was the spring of 1966 and the Middle East was poised for conflict with Israel in what would become the Six Day War. Three decades later, he wrote of an appalling assault he suffered in the semi-desert outside Tripoli. Sunbathing one afternoon, he looked up to find himself surrounded by "five Bedouin pastoralists", who took turns to rape him. The article, published in 1999 in Granta magazine, was entitled "Asking For It" and questioned whether a young white male, wearing only a pair of jeans in an Arab country at a politically unsettled time, had not indeed had it coming.

Neil Belton recalls that Hamilton-Paterson had spoken to him about the rape long before the article, and was struck by his "tough-minded response" to it. And Hamilton-Paterson can still remember "the smell of sheep's grease on their robes and the sand in my mouth".

Inevitably, people have speculated about Hamilton-Paterson's love-life. He never married, but did he want children? "I wouldn't have minded the idea of children", he says, "but not the actuality. Children would necessarily imply domesticity and I'm just not domestic." Frances Coady, the publisher, has known Hamilton-Paterson since the late 1990s, and admires his capacity to withdraw: "James combines the wondrous opposites of really being with and among people when he wants to be (he is frighteningly perceptive and quick), and then escaping totally and being fearlessly self-sufficient." Ann Wordsworth had detected a similar restlessness in Hamilton-Paterson at Oxford. "He was very easily bored, and sometimes he'd go silent and sort of go away into himself, switching off."

Hamilton-Paterson, for his part, maintains that he is just "not very good" at relationships. Though he discovered sex "well before Oxford" and "slept with everything and everybody", still he could not bring himself to live with anyone. "At a trivial level, I'm afraid you just get used to your own company, your own rhythms," he has said. Hamilton-Paterson has numerous friends, nevertheless, and remains deeply fond and admiring of the Suffolk writer Ronald Blythe, author of Akenfield . "Ronnie has been a great influence on me as a writer", he says, "not stylistically perhaps, but it was he who taught me to be absolutely professional and serious about my work."

By his mid-twenties, Hamilton-Paterson knew that he wanted to write. In 1966, he took on a job at St Stephen's Hospital in London as a porter and operating room technician. The job, which required him to act as a runner, carrying boxes of human tissue from one wing of the hospital to another, would help to fund his writing over the next two years. One winter afternoon he slipped and fell heavily while carrying a specimen box to the pathology lab. "The lid came off and there was a human stomach resting in the snow. I plucked it out, blew it off and stuffed it back in the box as if nothing had happened." The experience was enough to kill the "medical bug" he had inherited from his doctor parents. Between hospital shifts he wrote a children's novel, Flight Underground (1969), an adventure story with JG Ballard-like images of crashed planes and disused runways.

After his stint at St Stephen's, Hamilton-Paterson travelled through Vietnam and Brazil, teaching and writing journalism. In 1968, he was arrested by the Brazilian military in Manáos on trumped-up subversion charges, and later wrote an article for the New Statesman about his meeting with the remnants of Che Guevara's band in Bolivia. The article came to the attention of Paul Johnson, the magazine's then editor, who promptly invited Hamilton-Paterson to join his staff. Today, Johnson remembers above all the young man's punctiliousness. "There were three things I liked about James. First, he was totally independent-minded. Second, he had no obvious political affiliations (pretty rare in the high noon of Vietnam!). And third, he was a very accurate reporter indeed, who never knowingly perpetrated an error or an unfairness." Hamilton-Paterson was sent by Johnson to Northern Ireland to file on the Troubles there in 1969, and on various other "poorly paid" assignments round the world. He worked at the New Statesman for five years; his last full-time association with journalism was in 1975, as features editor of Nova, the (now defunct) women's magazine.

Since leaving Oxford, Hamilton-Paterson had continued to write poetry, and in 1974 he published Option Three (former White House jargon for "compromise"). The volume contained 12 poems written in Indochina, as well as one verse, "Poem for Ann", dedicated to Ann Wordsworth. "The assumption in those days was that James would definitely establish himself as a poet," says the cultural historian Mark Cousins, his friend for more than 30 years, "and certainly I don't think he then contemplated becoming a serious novelist." Dutch Alps (1984), his second verse collection, was a much more difficult and challenging work, the hermetic sparsities of the poems and their complicated rhythms showing a high-Modernist influence. Ann Wordsworth still marvels at their "sense of mortality and strange intimations of other existences", and points out that a sense of loss and foreboding intrudes on much of Hamilton-Paterson's later work. In Gerontius, typically, Elgar is portrayed not as the jingoistic old buffer of the Proms, but as a lovelorn soul in a more or less constant state of sorrow.

In 1979, in what turned out to be one of the definitive experiences of his life, Hamilton-Paterson discovered the Philippines. The country was the one place in South East Asia he had not visited at the time of the Vietnam war. An American helicopter pilot had told him: "Jim, when you next go on R and R, try Manila, it's so whacky." Hamilton-Paterson landed in the Philippine capital in 1979 on the day Margaret Thatcher was elected British prime minister. At first he disliked the city. "It struck me as a Coca-Cola-ised, post-South Pacific Asian re-creation of an American garrison town." Three hundred miles outside the metropolis, however, he discovered a "stunningly beautiful world" of sand and offshore reefs. He learned the local Tagalog language and came to love the people. Since 1979, he has spent a third of the year in a makeshift shelter on an islet, teaching himself to spear fish and feed himself.

Hamilton-Paterson's fictional début for adults, a collection of short stories entitled The View from Mount Dog, appeared in 1986 months after Imelda and Ferdinand Marcos had been ousted from the Philippines in a coup. In 1998, Hamilton-Paterson published his now classic account of the Philippines and the Marcos family, America's Boy, which sought to correct western misconceptions about the dictatorship. The book was admired in Manila, but some commentators found it glibly anti-American. British journalist Ian Buruma complained of the author's "tiresome taste for old-fashioned Third Worldism" and "anti-Washington bias". Hamilton-Paterson says he has never been under any illusions about the behaviour of a world power like the US. "Just as Washington used Haiti, say, as a bulwark against Fidel's Communist Cuba, so it used the Philippines as a strategic defence against red Vietnam."

In 1993, Hamilton-Paterson published his extraordinary novel Griefwork, about a lonesome botanist and glasshouse keeper. The book glows with scientifically precise descriptions of monsoonal and night-flowering shrubs, and is intensely atmospheric. "James is a wonderfully gifted amateur naturalist," says Lynn Barber, "and I daresay his love of taxonomies, genera and species is unique among contemporary British novelists." Griefwork is exquisitely written ("writing with a capital W", says Barry Humphries). "Even his emails are perfect," Barber comments.

In 1995 Hamilton-Paterson took time off to accompany a deep-sea salvage team in their search for two cargoes of gold torpedoed in transit during the second world war. His account of the expedition, Three Miles Down (1998), describes descents into murky depths off the west coast of Africa and feats of extraordinary physical daring. "James has always felt the need to test himself physically", says Cousins. "He enjoys the lonesome thrill of solitary competition and likes to measure himself against dangerous odds." The book laments the despoilation of the sea by pollution and overfishing. Clearly, Hamilton-Paterson is saddened by the idea of extinctions; but every species, he says, "must eventually disappear". Haunted by the evanescence of things, he has become, according to his admirers, one of our great prose writers of loss. "The sense of melancholy and loss was always there in James," Mark Cousins concludes, "and it's unassuageable."

James Hamilton-Paterson

Born: November 6 1941, London.

Educated: Windlesham House, Sussex; Bickley Hall, Kent; King's School, Canterbury; Exeter College, Oxford.

Career: 1966-68 St Stephen's Hospital orderly; '69-74 reporter for New Statesman; '74-75 features editor Nova magazine; 2000-02 science columnist for Das Magazin (Zurich); 2002 - science columnist Die Weltwocher.

Some books: 1974 Option Three; '86 The View from Mount Dog; '87 Playing with Water; '89 Gerontius; 1990 The Bell Boy; '92 Seven-Tenths; '93 Griefwork; '94 Ghosts of Manila; '95 The Music; '98 America's Boy; 2001 Loving Monsters.

Some honours and awards: 1989 Whitbread First Novel Prize (Gerontius);

· Cooking with Fernet Branca by James Hamilton-Paterson is published by Faber on June 3 at £10.99. To order a copy for £8.99 plus p&p call Guardian book service on 0870 836 0875.