The Oscar-winning film-maker Errol Morris made his name in 1988 with The Thin Blue Line, a bravura piece of documentary-making that gained the release from prison of an innocent man who had been on death row. But although he spent several years working on that investigation, it's not this crime that has maintained the most insistent hold on his intellect and imagination. That prize goes to one committed on 17 February 1970 at Fort Bragg in North Carolina.

He is not alone in his obsession. The killings that took place in the early hours of that morning and their protracted aftermath have cast an ever-lengthening shadow over not just America's criminal justice system – it is the longest-running criminal case in US history – but also its national media. A small library of books, a TV mini-series, countless documentaries and a forest of newsprint have all tried to explain what happened 43 years ago inside the home of Captain Jeffrey MacDonald, then 26 and a promising surgeon in the Special Forces.

At the heart of the crime and its coverage were capital letter concepts like Truth, Justice, Impartiality and Honesty but each one of these high ideals seemed to be in conflict with the others. Dishonesty was employed to establish truth, and justice often appeared less than impartial. In turn, all the media attention produced a seminal debate on the nature of journalistic ethics. Here are some basic facts on which all sides agree. At some time before 3.30am someone brutally attacked the MacDonald family in their home at 544 Castle Drive on what was then the open military base of Fort Bragg. MacDonald's 26-year-old pregnant wife, Colette, had both her arms broken and was stabbed repeatedly in the chest and neck with a paring knife and an ice pick. The couple's five-year-old daughter Kimberley was beaten across the head with a club and stabbed multiple times in the neck. Two-year-old Kristen was stabbed over 30 times in the back, chest and neck with a knife and an ice pick. MacDonald himself received relatively minor injuries except for a single stab wound that punctured his lung.

At around 3.30am MacDonald called the emergency services, while apparently drifting in and out of consciousness. He told the first military police to arrive that his family had been attacked by four intruders – two white men, a black man and a white woman. He said the woman held a candle and chanted "Acid is groovy" and "Kill the pigs". On the headboard in the marital bedroom the word "PIG" was written in blood.

America was a nation in turmoil in 1970. The backdrop to the murders was the Vietnam war, an increasingly vociferous protest movement and a drug-fuelled and disillusioned counterculture. Six months earlier Charles Manson's followers had committed similarly depraved murders in Los Angeles. At Roman Polanski's home they killed the director's pregnant wife, Sharon Tate, and with her blood smeared the word "PIG" on a wall.

That crime spooked the suburbs of America. It seemed to be the evil fruit of a society in flux, rootless, Godless, unpatriotic. And MacDonald was the very opposite of all that. A graduate of Princeton, he was the embodiment of the all-American hero – the athletic school quarterback who became a handsome, hard-working emergency doctor, the Green Beret who married his childhood sweetheart. Yet investigators soon began to suspect it wasn't drugged-up intruders who killed his family but MacDonald himself, who then carefully stabbed himself to make it look as though he had been attacked.

He was arrested by military police, but following an extensive inquiry, the longest pre-court martial hearing in military history, he was exonerated and honourably discharged from the army. He then made the tactical error of appearing on the Dick Cavett talkshow, the first of several disastrous interactions with the media. In front of a national audience, he complained at length about his treatment by the military authorities. Most viewers thought it suspicious that he spoke so much of his own plight and so little of his murdered family. Cavett talked afterwards of MacDonald's curious lack of "affect".

Watching the Dick Cavett show was MacDonald's father-in-law, Freddie Kassab, who up until that point had been his most tireless advocate. Almost overnight he became his most unstinting adversary. MacDonald made the further mistake of telling Kassab that he had tracked down one of the intruders and killed him. It was a lie, MacDonald later admitted, designed to get Kassab off his back, but it only served to deepen his father-in-law's distrust. Kassab relentlessly pressured the authorities to reinvestigate MacDonald, and five years later a grand jury was convened to establish whether the doctor should stand trial as a civilian. Eventually, in 1979, he was brought to trial in North Carolina. After hearing the evidence, the jury took just six hours to find him guilty. The judge, Franklin T Dupree, sentenced him to three consecutive life sentences.

The story might have ended there but for MacDonald's second fateful engagement with the media. Running up to the trial, he recruited Joe McGinniss to write a book about him and the case to help pay for his legal defence. McGinniss was a nonfiction author who had come to prominence with The Selling of The President, a bestselling book about Richard Nixon's 1968 presidential campaign. Both men were in their mid-30s, bright and sporty. They enjoyed each other's company and, although they had their own separate reasons for collaborating, by all accounts became close friends.

They signed a contract in which MacDonald would pick up a quarter of the author's profits. There was just one significant stipulation: McGinniss agreed to maintain "the essential integrity" of MacDonald's life story. So that he could have full access to his subject, McGinniss was made part of MacDonald's legal team. He moved into a fraternity house with MacDonald, his lawyers and various helpers. He then followed the trial assiduously. After MacDonald was found guilty and sent to prison, McGinniss remained in close correspondence, all the time giving the impression that he believed in his friend's innocence.

In 1983, almost four years after the trial, McGinniss published Fatal Vision, a big, closely researched nonfiction narrative in the tradition of Truman Capote's In Cold Blood. Beyond all else it was a damning indictment of MacDonald. Describing at great length the surgeon's bachelor lifestyle in southern California, where he moved after leaving the army, the book is replete with MacDonald's louche trappings: sports cars, Jacuzzis, a boat called the Recovery Room, and his various young girlfriends. It was a richly detailed character assassination in which McGinniss portrayed his onetime friend as a narcissistic, woman-chasing, psychopathic child killer. It was also a massive bestseller.

The following year MacDonald sued McGinniss for fraud and breach of contract. The resulting court case was held in 1987. There was a hung jury and to avoid an expensive retrial McGinniss's insurance agents settled out of court. That civil case formed the core of a two-part New Yorker piece by Janet Malcolm that was later published in book form as The Journalist and the Murderer. Malcolm viewed the relationship between McGinniss and MacDonald as a symbol of the unethical relationship that exists between all journalists and their subjects. The famous opening line has since become a defining comment on the duplicity of the news media. "Every journalist who is not too stupid or too full of himself to notice what is going on," wrote Malcolm, "knows that what he does is morally indefensible."

MacDonald and his lawyers went on to make a series of appeals against his murder verdict, arguing that his trial was unfair, that key evidence was suppressed and vital witnesses silenced. The most recent hearing was held last year in Wilmington, North Carolina, in which several key witnesses gave evidence before a judge. Around the same time two more books were added to the ever-expanding bibliography. McGinniss gave a summary update entitled, perhaps optimistically, Final Vision, and Errol Morris published A Wilderness of Error, which re-examines much of the evidence and finds it wanting. Morris first got interested in the MacDonald story in the 1980s, and has been dipping in and out of the details since the 1990s. In trying to explain what brought him back to the case again and again, he writes: "It wasn't the brutality of the murders. I was afraid of something even more chilling – that MacDonald was innocent. That he had been made to witness the savage deaths of his family and then was wrongfully convicted for their murders. I wondered if people needed him to be guilty because the alternative was too horrible to contemplate."

With diagrams, photocopies and timelines, Morris's book sets out to be a forensic analysis of the murders, the various legal cases that ensued, and the critical role of media coverage. The New York Times review said of Morris: "He will leave you 85% certain that Mr MacDonald is innocent. He will leave you 100% certain he did not get a fair trial." Moreover, it is written as a defence of the notion of verifiable truth, and against the fashionable encroachment of relativism. For Morris, truth is not a matter of subjective interpretation. Something happens or it does not. MacDonald either killed his family or he did not. And Morris is far from convinced that he did.

Errol Morris works in a large, airy studio in Cambridge, Massachusetts. When I visit him on a bitterly cold afternoon in February, he's editing a film about Donald Rumsfeld, provisionally titled The Known Unknown. Morris won his Oscar for The Fog of War, a brilliant documentary about Robert McNamara, the US defence secretary during the Vietnam war. Did he, I wonder, get access to Rumsfeld?

"Too much," he replies drily. "Eighteen hours of interviews."

Morris has cropped silver hair and a large genial face given to ironic expressions and knowing smiles. He was a private investigator before he was a film-maker, and before that he studied history of science under Thomas Kuhn, the man responsible for giving us the phrase "paradigm shift". Kuhn was what Morris would describe as a postmodern relativist. He was sceptical of the idea of objective truth, whereas Morris, ever since he was a child, has firmly held to the conviction that truth exists, even if it can't always be proved. Kuhn maintained that different paradigms, or scientific eras, were "incommensurable", meaning that they couldn't be understood in terms of each other. Therefore, he argued, it was a mistake to see the history of science as a progressive movement towards truth. But Morris pointed out that if paradigms were incommensurable, then the past was incommensurable, and any history of science was impossible. Kuhn grew so frustrated with his young charge that, according to Morris, he threw an ashtray at his head.

Morris was not deterred. As he later wrote in an essay on the subject of truth: "There is such a thing as truth, but we often have a vested interest in ignoring it or outright denying it. Also, it's not just thinking something that makes it true. Truth is not relative. It's not subjective. It may be elusive or hidden. People may wish to disregard it. But there is such a thing as truth and the pursuit of truth: trying to figure out what has really happened, trying to figure out how things really are."

It's this conviction, and the fact that so many people believe they know what happened without any real knowledge of the evidence, that drew him to the MacDonald murders. Harvey Silverglate, an attorney for MacDonald, is an old friend of Morris, and it was he who first told him of the anomalies, doubts and inconsistencies he had encountered with the prosecution's case. "The reason for me doing it," says Morris, "is that error is something worth examining in its own right. It's a subject of great interest to me. How we come to believe things that are either not true or could be false. The title of the book [a quote from Edgar Allan Poe] captures my thinking about this case and a lot of other things. It certainly has ramifications in a geopolitical realm. It's Iraq and Vietnam writ large to me, horribly destructive wars created in a crucible of false belief."

What Morris is really talking about is the manner in which we persuade ourselves that information is proof when it's merely information. He originally wanted to make a film about the case but he says no one was interested. He describes a meeting with a studio executive in which he was told that the film can't be made. "Why?" he asks. "Because he's guilty," the executive replies. "The man killed his family."

This is a common belief in America, and its prevalence, Morris contends, has much to do with the account disseminated by McGinniss. As Morris writes: "Jeffrey MacDonald was condemned to the story that had been created around him."

In reply in Final Vision, McGinniss coolly notes that Judge Dupree did not base his evidentiary rulings "on a point of view expressed in a book published four years later", adding that the conviction was also upheld in appellate courts well before Fatal Vision was published. In a series of emails with me, McGinniss was less circumspect about Morris's book, describing it as "a whoremongery, meretricious, unmitigated piece of shit".

The MacDonald case brings out strong opinions on either side. It's not just that those involved have spent years arguing their case, nor that they fervently believe they are the seekers and defenders of the truth – it's also a sense that those who disagree with them are guilty of bad faith. Although Morris insists he doesn't want to focus on McGinniss, he can't resist taking pot shots at his character. He talks disparagingly of McGinniss's memoir Heroes. "I find it an enormous irony," he says, "in those passages written by McGinniss in Heroes where he talks about his dreams of mutilating his family." He's referring to a diary section in which McGinniss records his guilt and anxiety about leaving his first wife and children for another woman, who would become his second wife.

"Morris uses this to try to compare me to MacDonald?" says McGinniss. "That's both pathetic and ludicrous. But if you are going to take this seriously, please note that I wrote, 'I dreamed terrible dreams about the maiming and destruction of my daughters.' I didn't dream that I maimed them."

McGinniss is correct but Morris's interpretation is instructive. So much of the MacDonald case comes down to who you believe, but in carelessly or intentionally misrepresenting McGinniss's meaning, Morris serves only to undermine his own credibility. And that presents an epistemological problem. If you don't have access to the original evidence, you have to rely on interpretation. But can we trust the interpreters? At which point do facts become shaped by opinion?

I ask Morris if he thinks McGinniss is a reliable witness. "What am I supposed to say?" he sighs. "It was shown fairly powerfully in court the enormous number of things that McGinniss lied about. And it goes well beyond the issue of whether he misrepresented his intentions to Jeffrey MacDonald, which of course he did. If this ultimately becomes a tribunal on whom to believe, I don't think we should believe anybody, including me."

Morris talks about the case as if it was a curse, a dark spell cast on everyone who becomes embroiled in its myriad complexities and conflicts. He lists the players whose careers have been shaped or stalled by its legacy and those who went to their graves believing the truth was drowned out by false testimony.

In contrast, McGinniss insists it's a simple, clear case deliberately muddied by a killer who refuses to accept his guilt. He also maintains that he was never sucked into the quagmire that MacDonald created. "The book was published in 1983, and I moved on," he says. If his reputation was affected by Malcolm's unflattering portrait of him in what has become a set text for journalists, he says he has never regretted his role. And having been excoriated by Malcolm, he claims he is little bothered by Morris's criticisms.

"Compared to a black widow spider like Malcolm, Morris is just a penny-ante grifter," he says. "Mostly, as I'm sure you know, Morris makes his living doing TV commercials for major American corporations. Nothing wrong with that, but his artiste pose is as phoney as a North Korean election. He writes about many perceived 'injustices' that occurred during the trial without letting the reader know that these were all litigated in later years and all decided in favour of the prosecution and Judge Dupree."

Despite their mutual antipathy, there is one area on which Morris and McGinniss share some common ground, namely Janet Malcolm's critique of journalistic deception. Morris rejects Malcolm's central accusation. "For Janet Malcolm [deception] becomes the crime of all journalism," Morris says, "but you could call it the crime of human existence if you chose. The crime is that McGinniss misrepresented his intentions to Jeffrey MacDonald, that he lied and manipulated, he betrayed. To me that's not the real crime."

Morris argues that Malcolm herself was guilty of a haughty indifference that in reality was no more elevated than McGinniss's ingratiating ruse. MacDonald sent her a "mountain of documents" during her research that she effectively ignored. In The Journalist and the Murderer she treats the whole question of analysing the evidence as a tiresome and futile inconvenience. "I know I cannot learn anything about MacDonald's guilt or innocence from this material," she wrote, without having examined the contents. "It is like looking for proof or disproof of the existence of God in a flower."

"You can't be interested in the meta-history of anything without ultimately being interested in the history," Morris insists. "That's my ultimate criticism of Janet Malcolm, whom I revere, that she wants it both ways. You can't write about the relationship of a journalist and a murderer, and somehow avoid the underlying reality of whether or not… he did it!"

He draws a comparison between Malcolm and McGinniss that makes them both equally culpable of abandoning MacDonald to his hopeless fate. "It's like the parable of the drowning man. What would you prefer, the person who says 'Fuck you, I'm not going to throw you a life preserver because I think you're a cold-blooded killer and psychopath, a misogynist and a narcissist' or the postmodernist standing there with a life preserver who says, 'You just misunderstand the nature of our relationship. You just see yourself as a drowning man and me as a person standing on the shore with a life preserver, but I'm really examining the relationship between a drowning man and the person standing on the shore with a life preserver, and because of those constraints, I really can't do anything'?"

Ever since the murders, MacDonald's best hope of a life preserver has been a woman called Helena Stoeckley. Now dead, like many involved in the case, she was a major drug user living in Fayetteville near Fort Bragg who confessed to being at 544 Castle Drive the night of the murders. When the military police arrived at the house, MacDonald described the female intruder as wearing a blond wig, floppy hat and white boots. Racing to the scene, a policeman had seen a woman of that description on a street corner not far from MacDonald's house – a suspicious sight at 4am. Stoeckley admitted to being at the house, and over the years repeatedly confirmed her presence there. But at other times she withdrew the confession, and also said that she had an affair with MacDonald and had tried to buy drugs from him – both of which claims have been dismissed by all parties.

One of those to whom Stoeckley allegedly confessed was a federal marshal called Jimmy Britt. Twenty-five years after the trial Britt suddenly came forward with an affidavit stating that he had driven Stoeckley from Greenville in South Carolina to Raleigh in North Carolina, a five-hour journey, to attend the trial. During the trip, he said, Stoeckley told him she was in the MacDonald house while the murders were committed. He also stated that he had witnessed a prosecutor named James Blackburn, in a pre-trial meeting, threaten Stoeckley with prosecution if she testified that she had been at Castle Drive. Blackburn denied coercion and insisted that Britt was not present. In the event Stoeckley denied in court (although not in the presence of the jury) that she was at the MacDonald house at the night of the murders.

In A Wilderness of Error Morris writes: "The significance of Stoeckley's testimony and the outcome of the trial itself depends on whom you believe: Blackburn or Britt." But at the hearing in Wilmington last year it was revealed that Britt did not and could not drive Stoeckley from Greenville because that job was done by several marshals and there is conclusive proof that Britt was not one of them. In other words, he lied on his affidavit. If he lied about the journey, was he also lying about seeing Blackburn threaten Stoeckley?

Morris's book came out too late for him to amend, and one harsh review dubbed the result The Wilderness of Errol. When I ask whom he now believes, Blackburn or Britt, he begins to sound like the sort of relativist he derides. "Do I believe the various challenges made by the prosecution to Britt's account invalidates everything he said?" he asks rhetorically. "No, I do not agree. Do people remember things that didn't happen and forget things that did happen? All the time."

Blackburn, an assistant US attorney, was disbarred in 1993 for ethical violations and jailed, having been convicted of embezzlement and fraud. Understandably, Morris raises question marks about his character and reliability. But it also transpired that Britt, the marshal, was embittered about his government service, and it seems likely, according to evidence presented in Wilmington, that he knew that certain government records were to be destroyed after 25 years – hence the timing of his affidavit.

The MacDonald case has more than its fair share of oddities. Every established truth appears to rest on a foundation of doubt, and each of the dramatis personae embroiled in controversy. One of the most controversial figures was Judge Dupree. Even McGinniss allowed in Fatal Vision that Dupree demonstrated a physical, if not judicial, bias against the main defence lawyer, the Californian Bernie Segal. "… from the earliest days of the trial," he wrote, "the expression most often seen upon [Dupree's face] as Bernie Segal conducted cross-examination was one of distaste."

Segal believed that Dupree, a southern gentleman of the old school, was antisemitic. But perhaps he just didn't like Californians. He certainly did MacDonald few favours, ruling against the admission of psychiatric evidence that suggested he was not the kind of man to commit such a horrific murder, and preventing witnesses who heard Stoeckley confess from giving testimony.

The figure of most controversy, though, is Jeffrey MacDonald. He remains incarcerated and steadfast in his protestations of innocence. In a prison ceremony in 2002 he married a woman who is equally convinced that he is not guilty. Yet nothing adds up with him. His story about the hippie intruders seems fantastic. Acid is groovy? Kill the pigs? That sounds like dialogue from one of Roger Corman's more hamfisted exploitation movies. And why would a gang go to such sustained and ferocious lengths to kill a pregnant woman and two small children and yet leave a powerful and therefore dangerous man with only a single puncture wound?

"It's very interesting these syllogisms we set up in our head," says Morris. "We think we're arguing to some incontrovertible logical conclusion but we're doing nothing of the sort. "Why is MacDonald alive and his family dead?" asks Morris. "Good question. To me, you turn it around: Are you saying because he's alive, he killed his family?"

Equally, why would a man who had never previously shown aggressive tendencies, known as a loving husband and father, without warning slaughter his own family? The prosecution was never able to construct a convincing motive. Nor was McGinniss. Instead he assembled a large array of character assessments and examples of minor indiscretions to create a portrait of a seemingly normal man who, underneath a carefully composed exterior, was a raging psychopath.

McGinniss discovered that MacDonald had been taking an amphetamine diet pill called Eskatrol and surmised that the drug, and a punishing work schedule, triggered his latent psychopathy. Morris is scathing of McGinniss's account of that February night.

"MacDonald comes home and he finds that one of the kids has wet the bed and he goes berserk because he's on Eskatrol, and the underlying narcissism/misogyny/psychopathy/sociopathy caused him to slaughter his entire family? My wife is actually very good on this. We've all been tortured by this case. She says MacDonald's an emergency room doctor. Are we to believe he's never seen bodily fluids, blood, urine, God knows what else? A Green Beret emergency room doctor sees the bed is wet, 'Oh no, bodily fluids, I think I'll kill everybody.'"

Morris has had several communications with MacDonald, although he avoided contact during the writing of the book so as not to allow emotion to affect his focus. "He is a problematic person," he says. "Is there anything he said to me, anything in his demeanour that tells me he slaughtered his family? Not really, no. To me the issue is not whether I like Jeffrey or I don't like Jeffrey, because I do like Jeffrey, but I still think it's irrelevant. There are lots of people from 1969 and 1970 who really liked him, admired him and his family, saw him as a loving father."

Unlike Morris, McGinniss interviewed the jurors at the trial and he says they also warmed to MacDonald. "Not one was gleeful," he says. "They all liked and felt sorry for MacDonald. But they also all knew he'd committed the murders, and so, in many cases teary-eyed, they did their duty as citizens."

Ultimately what did for MacDonald was not his personality or his lifestyle but the forensic evidence. Morris maintains that the crime scene was despoiled by the military police who first arrived and that therefore the forensics are unreliable. But every disputed criminal case of note has its totemic piece of evidence. With OJ Simpson it was the glove that didn't fit. In the assassination of JFK it was the so-called magic bullet. In the MacDonald case it's a pyjama top.

Colette MacDonald was found by military police with her husband's folded-up pyjama top placed over her chest. MacDonald said he took the top off and laid it on his wife after he tried and failed to revive her. But the top had 48 ice-pick holes in it and MacDonald had only one puncture hole in him. He explained this by saying the top had been pulled over his head by the assailants as they tried to stab him with the ice pick. In court the prosecution showed that the top could been folded in such a way that its 48 holes could be aligned with the 21 wounds in Colette's chest, suggesting that MacDonald had placed it there before attacking his already unconscious wife. Morris argues that the prosecution withheld evidence that showed that the directionality of the pyjama fibres did not match up with the hypothesis demonstrated in court.

"I believe the pyjama top was deliberately misleading," he says. "There was a systematic pattern of the prosecution manipulating evidence, withholding evidence, destroying evidence and hiding evidence."

One of those prosecutors, Brian Murtagh, sent me a long email explaining exactly why the pyjama top demonstration was accurate and conclusive. Trying to work out who was right and what was real, I began to understand why Malcolm was reluctant to examine the evidence. The case can resemble one of those images that depict two entwined but distinct faces, only one of which can be seen at a time.

To believe one narrative is to reject the other. Yet only one can be true. There is a wilderness of error or a wasteland of truth. Either MacDonald has spent 34 years in jail for a crime he did not commit or he is a manipulative psychopath shamelessly asserting his innocence. Neither version is palatable.

"I have a slime theory of humanity," Morris told me with a playful grin. "Like molluscs, we leave this trail of slime behind us that really prevents us from seeing the world. Our own desire to create narratives, to suppress certain kinds of observations in favour of others. Our failure to collect evidence that may interfere with our underlying core beliefs. Whatever. That by the time we're done we've obscured the world around us in such a way that we can never recover it. It's not that the truth is unknowable, we make it unknowable. Basically, you can think of it that we defecate on reality."

Later this year the judge in Wilmington is due to report on his findings on the long saga of the people versus Jeffrey MacDonald. It should be the last throw of the dice for the 69-year-old doctor. McGinniss and Morris are of the same opinion: they are not expecting him to get lucky.

A Wilderness of Error is published by the Penguin Press in the US