In the course of their famous book-length interview, François Truffaut once asked Alfred Hitchcock about his approach to literary adaptation, and Hitch's response was as magisterial, worldly and mischievous as one would expect: "What I do is to read a story only once, and if I like the basic idea, I just forget all about the book and start to create cinema. Today I would be unable to tell you the story of Daphne du Maurier's The Birds. I read it only once, and very quickly at that."

Hitchcock's comment was the first thing that occurred to me when, towards the end of last year, I was approached with an interesting proposition. "From Page to Screen" is the name of a small film festival which takes place in Bridport, Dorset. It's now in its third year and, as its title suggests, is dedicated exclusively to films which are adapted from literary sources. This year, for the first time, the organisers decided that they wanted to invite a guest director to oversee the programme.

I accepted the offer at once, and then almost immediately wondered what I'd let myself in for: because the truth is that 99 times out of 100, I'm with Hitchcock on this one. Draw up one of those faintly ludicrous but fascinating lists of the 20 greatest novels, and then do the same for movies: do they match up, at all? Of course not. Joyce's Ulysses might well be on the first list, but Joseph Strick's Ulysses (1967) certainly won't be on the second. Pride and Prejudice could possibly be on the first, but neither Robert Z Leonard's nor Joe Wright's adaptations will make the second. And none of these examples is a travesty, exactly, although we could all name some of those if we wanted to: film history – especially recent film history – is littered with examples where a good novel has been transformed, not into an average movie, but an outright disaster: Captain Corelli's Mandolin and The Bonfire of the Vanities spring immediately to mind.

Looking a little more closely at what Hitchcock said gives us a clear explanation of why this is so often the case. The question Truffaut specifically put to him was whether he would ever consider making a screen adaptation of a great novel such as Crime and Punishment. To which the director answered: "Well, I shall never do that, precisely because Crime and Punishment is somebody else's achievement. And even if I did, it probably wouldn't be any good." "Why not?" Truffaut asked. "Well, in Dostoevsky's novel there are many, many words and all of them have a function." "You mean that theoretically," Truffaut prompted, "a masterpiece is something that has already found its perfection of form, its definitive form." "Exactly," Hitchcock answered, "and to really convey that in cinematic terms, substituting the language of the camera for the written word, one would have to make a six to ten-hour film. Otherwise, it won't be any good."

This point might seem to be obvious, but it holds good: any two-hour feature film which attempts to render, in cinematic terms, the full complexity of a serious novel-length work of fiction is almost certainly doomed. I suppose that's why some of the most satisfying screen adaptations have been television serials – from the definitive 1995 BBC Pride and Prejudice to the one that looms over them all, ITV's Brideshead Revisited – which allowed itself a luxurious 659 minutes to cover the novel's 350-odd pages, and with its unapologetic reliance on voiceover devolved, at some points, into little more than a recital of the printed text with accompanying footage of country houses and fey young men in white linen suits.

The more I thought about it, however, the more determined I became to truffle out – for my own satisfaction, if nothing else – some examples of the rare celluloid exceptions: occasions when a demonstrably fine literary work has been adapted into an equally fine piece of cinema. Could such prodigies really be so hard to find?

Looking back over the last few months' most high-profile releases, it's striking to see how often film-makers still turn to the contemporary or recent novel for material. We have had films based on Joe Dunthorne's Submarine, Murakami's Norwegian Wood, Greene's Brighton Rock, Ishiguro's Never Let Me Go, Mordecai Richler's Barney's Version and Charles Portis's True Grit – with the Coen brothers going out of their way, in this case, to locate their film as an adaptation of the novel rather than a remake of the John Wayne version. 127 Days and The Social Network were also adapted from literary sources, albeit non-fiction ones. Clearly, few people in the film business nowadays share Hitchcock's reservations about literary adaptation, or would subscribe to Truffaut's caveat that the safest works to adapt are "popular or light entertainment novels".

Barney's Version, to take just one of these examples, might stand as a textbook demonstration of the pitfalls as well as the rewards of transferring literary material to the screen. It is an entertaining, funny, well-crafted mainstream film. It broadly adheres to the narrative highlights of its source material. It is obviously made out of love and reverence for Richler, both as a writer and as a Canadian public figure. (Friends and younger members of his family appear in walk-on roles.) And yet there is no getting away from the fact that it is radically unfaithful to the tone and the narrative strategies of the original novel.

To take the tone first: Richler's novel is, above all, bracingly and relentlessly unsentimental. Although there is a love story at its core, it is a love story hedged around with bitterness, misunderstanding and betrayal. Barney Panofsky first glimpses the love of his life, Miriam Greenberg, at his own wedding to another woman. After talking to her for just a few minutes he confides to his best friend that "For the first time in my life I am truly, seriously, irretrievably in love" – a comment overheard by his bride, who naturally thinks that he's referring to her. Minutes later Barney has left the wedding and is sprinting along to the railway station in pursuit of his new true love.

All good romcom material, on the surface – which is exactly how the film plays it. But on screen, it's too soft around the edges: Barney's flight is presented as a mad, impulsive, adorable gesture, rather than (as in the novel) the crowning instance of his genius for messing up his own life and the lives of those around him. The filmmakers don't dare – or cannot afford – to make their hero unsympathetic or unlovable, whereas in the novel he shuns the reader's sympathy at every opportunity, in baroque displays of misanthropy and self-loathing: "I dislike most people I have ever met, but not nearly so much as I am disgusted by the Rt Dishonourable Barney Panofsky."

Rather than registering and intensifying this note of bitterness, the film mellows still further as it proceeds. The shot of Barney's grave at the end of the film, bathed in autumn light; the winsome, regretful smile of Miriam as she visits it; the brushes of Rachmaninov-lite on the soundtrack – all of these, one can't help feeling, would have the novel's narrator reaching for his sick-bag. Nevertheless, we can understand the commercial imperatives that dictate them. The way that the film misses out on all the layered complexity of the novel's narrative, though, is a more fundamental matter of one artistic form failing to accommodate another. As the title of the novel makes clear, Barney's version of events, as put forward in the novel, is just one interpretation among many. The fact that, as he writes, he is suffering from Alzheimer's (which in the film is used to wring yet more sympathy out of us) Richler sees as a dark joke which adds another level of unreliability to his account. Corrective footnotes inserted by his son may or may not be there to set the record straight. And most important of all, "Barney's version", in the novel, is meant to be a riposte to another book supposedly written by his arch-rival Terry McIver. McIver is a former friend who has gone on to become an acclaimed, garlanded and terminally dull pillar of the Canadian literary establishment. His own self-serving autobiography, we are told, "gratefully acknowledges the assistance of mediocrity's holy trinity: the Canada Council, the Ontario Arts Council, and the City of Toronto Arts Council". In this way, Panofsky's own fictional memoir takes its position not only on Canada's literary battleground, but in a war between two opposing worldviews: one of them prissy, respectable and politically correct, the other one sour, sexist, pugnacious but at the very least honest and clear-sighted. Barney's Version the novel is a passionate and scabrous polemic arguing for the primacy of one kind of narrative voice over another. The film, meanwhile, is a pleasantly teary soap opera about a failed marriage.

Richler's long, complex, tricksy novel is probably unfilmable: that is to say, it's a construct which in Truffaut's phrase "has already found its perfection of form, its definitive form", and any attempt to transfer it to another medium is doomed to do no more than skim across the surface. Perhaps, then, in order to find more nearly faithful adaptations, we need to go to the other end of the literary spectrum and look at shorter works which achieve their effects by a kind of brilliant limpidity. And the first example which springs to my mind is James Joyce's story "The Dead", in its exceptional screen version by John Huston.

Huston directed the film from his wheelchair in the last year of his life. He had already proved that he could, on occasion, produce masterly adaptations: The Maltese Falcon, The Treasure of the Sierra Madre and The Man Who Would Be King are all exemplary. If his version of Moby-Dick fails, that merely testifies to the unwieldy nature of his source material – and indeed, with "The Dead" he found himself faced with the opposite problem, because Joyce's story did not quite provide enough narrative material to fill up a feature film's worth of screen time, obliging the adapter (Huston's son Tony) to introduce a new character called Mr Grace who recites a long Middle Irish poem which has no equivalent in the original text.

The story itself is an odd, beautifully misshapen thing. Roughly 40 of its 50 pages are devoted to a detailed description of the annual party given by Kate and Julia Morkan, two elderly mainstays of the Dublin musical world. It is the first week of January, 1904, and snow lies on the ground. The description is mainly filtered through the consciousness of the Morkans' nephew Gabriel, as he prepares to deliver an elaborate and over-rehearsed homily to the guests at the supper table. Pages and pages of musical chit-chat and incidental detail are lovingly rendered before we get to the nub of the story: the revelation that Gabriel's wife Gretta has been nursing memories of a secret romantic tragedy from her early life, brought to the surface this evening when she overhears snatches of an old folk melody from western Ireland.

Huston follows this peculiar narrative contour with absolute fidelity. The song – triggering Gretta's memory and Gabriel's moment of epiphany – appears almost exactly one hour into the film's 80 minutes' running time. In the 60 minutes which precede it, the film has no conventional narrative momentum whatsoever: a fact which didn't strike me the first time I saw it, in the year of release, but which on a re-viewing, 20 years later, felt like an extremely bold (and necessary) artistic decision: also a tiny shard of evidence – but to me a compelling one – that commercial film-making has become more and more formally conservative in the past quarter-century.

The last few minutes of the film follow Joyce's final pages closely but not exactly. Fragments of dialogue are transposed, a funny story Gabriel tells at the party being cleverly turned into a ruefully futile attempt to stir his wife out of her melancholy silence during their cab journey home. The painful conversation between husband and wife in the hotel room is just as Joyce wrote it, and flawlessly played by Donal McCann and Anjelica Huston. Then comes the celebrated final monologue, for which the film slips into voiceover for the first time. The script truncates and rearranges it, but holds to its tenor and spirit. As McCann's voice unfolds, the screen offers us simple shots of wintry landscapes at dusk, the folk tune recurs, distantly, on a solo clarinet, and we are treated, for a few overwhelmingly moving moments, to what film can and should but rarely does become: a perfect counterpoint of word, music and image.

Part of the emotional impact of that sequence, certainly, comes from the knowledge that it was the last thing Huston would ever shoot. And part of its fascination comes from the collision it represents: a "classic" Hollywood director in the twilight of his career paying homage to a passage in literature which represents, as much as any other, the birth of modernism. It was Joyce himself – the coincidence is almost too neat – who opened Ireland's first cinema, the Volta Cinematograph, and foresaw that this new medium must change the way that books were written for ever. From the early 1900s onwards, literature and film became storytelling bedfellows, and it must be for this reason that nearly all the best adaptations are of modern books, while attempts at doing the pre-20th-century novel on celluloid usually end up as mummification rather than reinvention. (As always, there are exceptions: I have a soft spot for Tony Richardson's Tom Jones and John Schlesinger's Far From the Madding Crowd, although these are both really swinging sixties romps which happen to be played in period costume.)

Joseph Losey's Accident, for instance, presents another example of modernist literature and cinema conjoining as if made for one another. Nicholas Mosley's novel, published in 1965, must have seemed gleamingly strange and original at the time: today it feels even more so. (Writing as well as film-making may have got more reactionary over the years.) Harold Pinter adapted it and wrote probably the best of his produced screenplays. Here, in fact, we seem to have a case of novelist and adapter so in sync with each other that a sort of symbiosis starts to operate. The thought and speech-rhythms of Mosley's central character Stephen, an Oxford philosophy don – so meticulously rendered in the novel – seem to have seeped into Pinter's consciousness and can still be heard echoing through the pretentious verbal sparring of Hirst and Spooner in No Man's Land seven years later. Moreover, Pinter appears to be so tuned in to Mosley's cadences that he is able to transpose some of his dialogue verbatim and then improve upon it, Pinterising it in the process. Mosley has one of his dons remark on a survey of students at Colenso University, Milwaukee, which shows that 70% of them have sex in the evening, 29.9% in the afternoon and 0.1% during a lecture on Aristotle. In Pinter's version, this is followed by a bored silence, broken only by the Provost observing laconically: "I'm surprised to hear Aristotle is on the syllabus in the state of Wisconsin."

As with The Dead, we might look at Accident now and feel our jaws drop at the thought that there was once an era when something so unashamedly highbrow could attract commercial financing and receive a wide release. The film makes no concession to its mass audience in rendering the oblique, rarefied atmosphere of Mosley's novel: the single most surprising thing about it, now, is that it allows itself to preserve all of the original's many ambiguities. And this, perhaps, suggests a reason why, even though modern literature and cinema could easily form a natural partnership, the marriage nowadays is so rarely successful. So much of the best modern fiction tends towards ambiguity and open-endedness, while increasingly the commercial cinema has a fetish for closure and ends neatly tied.

This might lead us to a partial explanation for the disappearance into near-oblivion of what I consider to be one of the best adaptations of a modern novel ever made. Marilynne Robinson's Housekeeping was published in 1981 and, 30 years on, has already achieved the status of a minor classic. Bill Forsyth's film version, made in 1987, is an unswervingly faithful adaptation, preserving the narrative shape, the tone, the desolate backwoods atmosphere, even finding visual correlatives for Robinson's scriptural, luminous prose. And yet it has been almost completely forgotten. It's never been available on DVD, and none of the Robinson fans I've spoken to recently, either in Britain or America, seems to be aware of it.

The film stars Christine Lahti as Sylvie, a wanderer and free spirit who finds herself, through a series of family tragedies, summoned back to Fingerbone, Idaho, to look after her orphaned nieces, Ruth and Lucille. The novel is narrated by Ruth, in the first person: Sylvie may be the main focus of attention but we see her always from the child's wondering point of view. Naive, trusting, optimistic, Sylvie herself is almost as childlike as the girls she has been charged to look after, and the two sisters react to her in different ways. The conformist Lucille, the younger of the two, finds her erratic behaviour an embarrassment, while for Ruth she becomes a sort of role model, a symbol of how life might be lived more freely and intensely than she had thought possible.

One of the unusual things about Housekeeping the movie is that it's an adaptation made by someone not previously known for adaptations – in this case, a writer/director of original screenplays who had already staked out a highly personal territory. Forsyth became famous for his second feature, the charming comedy Gregory's Girl, in the early 1980s (his affinity with young actors well in evidence even then) before scoring a considerable commercial success with Local Hero, which was followed by the more introverted and melancholy Comfort and Joy. Housekeeping, the first fruit of his troubled American career, came as a surprise to most of his admirers. His first four films had established him as something unusual in British cinema – a genuine auteur, with a distinctive tone and point of view – and yet this project required him to submerge his own creative voice, putting himself entirely at the service of another artist's vision.

He seems to have done this quite willingly, with a commitment born out of passionate admiration for Robinson's novel. At just over 200 pages, with a broadly linear narrative, the book doesn't pose the same kind of structural or compression challenges that something like Barney's Version throws up. Forsyth uses voiceover, but uses it sparingly: its point being to establish Ruth as the "author" of the narrative, rather than to allow copious quotation from Robinson's prose (tempting though that may have been). Loose-limbed, intimate, rigorously economical in its dialogue and its storytelling, Housekeeping deserves the highest compliment an adaptation can attract: it doesn't feel like an adaptation at all. It doesn't feel "literary". Its most magical sequence shows Ruth (playing truant from school) setting out with Sylvie to explore a secret place her aunt has found – "stunted orchard and lilacs and stone doorstep and fallen house, all white with a brine of frost" – and then spending a whole night out on the lake. Of course, you don't get any of the "many, many words", judiciously chosen and crafted, with which Marilynne Robinson paints this scene, taking in such borderline-surreal details as the twigs of apple trees festooned with marshmallows which Sylvie has placed there in order to entice wild children out of their mountain hiding-places. Instead, you get the British Columbian landscape photographed in all its dappled beauty by Michael Coulter, you get Mike Gibbs's eerie, subtly dissonant music (scored for strings only), and the utterly truthful, unaffected performances of Lahti and the young Sara Walker. And for once, it feels like a fair exchange.

The novel ends – if I'm reading it correctly – on a note of poised ambiguity. Having alienated the townspeople, Sylvie and Ruth make a daring escape by nightfall across the narrow, precarious railway bridge which extends for miles across the lake. Do they make it, or do they die in the attempt? Robinson does not quite let us know, and neither does Forsyth, ending his film on the dark, tantalising image of the two fugitives setting forth on to the bridge, their figures dwarfed by the parallel lines of track receding ahead of them into infinite blackness. It's a comfortless but arresting image, typical of a film which – like the novel – refuses classification either as comedy, tragedy, or anything in between.

There lies its greatness – and there, as far as Hollywood is concerned, lies its failure. In an unhappily prophetic 1985 interview, Forsyth – then scouting locations for the film – reflected that, with loose cannons like him, studios were always going to "worry that you are going to get involved in something that is unwatchable or, worse, unmarketable. Unmarketable is a much more worrying term for them." Housekeeping's disappearance from the collective memory bank of most film-goers proves nothing except that, in this case, Forsyth did make a truly unmarketable film. But he also did the right thing. He honoured his source material; and incidentally proved that even Hitchcock – now and again – could be wrong.

From Page to Screen is at Bridport Arts Centre, 13-17 April. The Dead and Housekeeping will be screened on 14 April, followed by a conversation with Bill Forsyth. Accident will be screened on 15 April, followed by a conversation with Nicholas Mosley. www.bridport-arts.com