Back in their day, the Teale brothers were like a pop group. "I'm not being funny," Alfie, 72, the eldest, says. "We dressed up. A lot of people loved us." And you can see it now in the sharpness of their suits, the dapper suede shoes, the red hanky in David's top-pocket. Strong chins, piercing eyes, faces full of furrows, they are peas in a pod, hard to tell apart, a three-piece: easy.

They've got the gift of the gab between them too. "Where'd I live? Why? You coming round to my place?"

They give me a crash course in cockney slang, Holborn branch. "Richard" (the Third) is a bird (of the female variety). "Rory" (Amor) is the floor. If someone tells you there's "a melt on your landslide" it means there's an ice cream, a geezer, a face, to one side of you.

Then, quietly, Bobby, 70, the middle one, says: "I don't know it." The other two stop larking about. Alfie puts his briefcase on the floor. David, the youngest at 69, looks like he wants to make a joke, but can't think of one. Bobby has been drinking Coke, not tea like his brothers. His glasses have a light-sensitive tint. He has better teeth. Now he opens his hands wide and holds them there, as if measuring an enormous gulf. His tone, with its mid-Atlantic edge, is full of sorrow, regret, a touch of defiance. "I don't know the lingo."

The tale of the Teales is an extraordinary one, spanning four decades, from 1960s gangland London, to a small town in Utah, USA. On the surface, it is a story of the Krays, and their hold on an entire community. It takes in police corruption and bribery, a world emerging from the ashes of the second world war, where criminals rubbed shoulders with politicians, showbiz dined with murderers, where no one could be trusted. But it is also a story about a family torn apart, about betrayal and loyalty and love – and, at the centre of it, a man who decides at the age of 67 to learn to read and write and in so doing tracks down the beloved brother he had thought for 45 years was dead.

Growing up, they couldn't have been closer, the Teale boys. The middle three in a family of seven, they were war-babies, Alfie born "the night Hitler bombed Lamb's Conduit Street". Their childhood was spent ducking and diving, nicking flowers from Covent Garden when they were late for school to stop the nuns at St Joseph's in Macklin Street from tanning their hides. They shared a bed. "Alfie would say 'Witches' trick,'" Bobby remembers, "and we'd all have to turn over at the same time." Skipping secondary school, they wheeled and dealed (like their old man) down Petticoat Lane, stole bikes and shop-lifted, and did a bit of time in reform school.

Bobby was making a go of things – he had a little boat business in Shanklin, Isle of Wight, and a wife and baby – when the Krays first came on the scene. But his brothers would visit, full of stories of these lads, as Alfie says, "this smart firm of people who could handle theirselves". David and Alfie, street-traders by day, were doing a bit of this, bit of that by night for the Krays – smuggling the odd gun into a club, but mainly entertaining with their jokes and their poetry. Before long, after using his boat to do a mid-Solent drop ("Was it a body? I wasn't going to ask"), Bobby had been drawn up to London too. Reggie paid for his divorce. The Krays had taken over their mother Ellen's speakeasy, the 66, in Islington. "And they were happening," Bobby says now. "They were smart. We were smart. As soon as we came connected we'd get the same respect as they got. Suddenly it was like, phwoah – this is good."

"You knew they wasn't normal people," Alfie continues. "But you didn't think they was evil people because we certainly wasn't evil people. Ronnie Kray said to the Firm, he said, two or three times, 'These young boys, they work for their living. We don't have to pay them.' Some geezer barred David and me from a club because David kept pulling all the cigarette girls. Yeah you," he points at David, who is making a hang-dog face. "I told Ronnie and he went down there, 'Don't you dare bar these two boys, they're hard-working boys.'"

It was glamour and parties, Judy Garland and Talk of the Town, Lord Boothby and El Morocco. And then the violence started – shootings, hits, casual, mindless brutality. Alfie says: "I've been thinking very seriously about Violet Kray, who I adored. Lovely lady. They couldn't have loved their mum that much to be able to fetch murders on her doorstep." Both David and Bobby were raped by Ronnie, who was openly gay but had a propensity for men who weren't, though they didn't tell even each other until last year. The police knew, but didn't use it in court. When Ronnie called David a liar, David says: "I looked him right in the eyeballs and said, 'Truth hurts.'"

The night Ronnie killed rival gang-member George Cornell (one of the murders that was to take the Krays down), both twins and half the Firm fetched up in David's flat and didn't leave for two weeks. It was at this point that Bobby, worried for the safety of David's young family and his own 11-year-old brother, Paul (whom Ronnie appeared to be grooming), became a police informer. It took three years, and the arrival of Nipper Read, for his evidence to be used in court. In the meantime, the Teale boys had spent almost as long in prison, for a blackmail charge they deny to this day.

Bobby never told his brothers that he had "grassed". All three gave evidence against the Krays at the Old Bailey in 1969. (Alfie and David did a deal to get their mother off a robbery charge, though that's another story.) All three were given protection against reprisals. But a wedge had been driven between them. Bobby wasn't himself. "I couldn't understand it," David recalls. "Your brother don't talk to you; you think you're going off your head." In 1969, convinced that his life was in danger as much from Scotland Yard – "I knew a ton" – as from the Krays, Bobby visited his "lovely mum" for the last time ("the pain was like a bullet in my heart"), took the boat-train to France and went on the run. "Someone in the police told me I should stay away for five years. Call it psychotic paranoia, whatever, but I thought I had to get out now."

David picks up the story. "I was told he was dead. I took my mother up to the police station, maybe 10 times. I'd say, 'Let's find out about Bobby,' going backwards and forwards. I'll be honest with you. It was heartbreaking for my mother. After about three years they said, 'Assume he's dead, we've got no trace on him, no nothing,' words to that effect." He breaks off, stares into the middle distance. "Gutted."

"I spent three years with my mother while she was dying," Alfie puts in. "Me and my wife. I had the hairdresser in every week for her to look nice. She died ... oh, I can't remember the date offhand."

"2004," Bobby says, almost inaudibly, his eyes on the table. "I watched her fade away. She had a stroke, in and out of dementia. She used to say, 'Anyone seen Bobby?' And one day, I said, 'Mum. I've got beautiful news for you. Bobby's all right. He's coming back.' 'Oh, lovely.' Total fabrication. And the funny thing about it was, she seemed to get a bit better."

In the years after Bobby's disappearance, Alfie, a good Catholic, stayed put, living in Holborn, married to Wendy, working the streets: "Oxford Street, Regent Street, Tottenham Court Road, stalls, swag shops, cash'n'carries, that sort of thing."

David's life was more colourful – he traded antiques in Amsterdam, and "got involved with diamonds" in Sierre Leone about which, if he wasn't advised to stop, he would talk at great length. In 2010, he was helping people with drug problems, alongside a teacher called Lali. One day he was telling her stories about his past (Charles Taylor, the Krays). "And she said, 'You wanna write a book.' And I said, 'Lali, between you and me, I can't read or write.' She got me on to this course and one of the first things I did was get on a computer at the library. Fantastic, when you think of it. Then I said, 'Lali, how do I find out if my brother's dead or alive?' And she said Facebook."

Bobby had spent time in Gibraltar, then travelled to America. He went into construction, married a Canadian nurse and had three children, still on the run in his head – "Always doing the street-smart thing, always looking, always observing." He tried to bury his past.

"I blanked my family from my mind – 10, 20, 30, 35 years. I was conditioning myself that I would die without knowing." Then a few years ago, his daughter gave him a picture book for Christmas, Defining Moments in History, and in it was a picture of the horse-drawn hearse at Reggie Kray's funeral. A huge torrent of emotion welled up in Bobby, of anger and grief, and he found himself having to explain to his children, who were baffled by his response to the book, his personal knowledge of the Krays, and a little bit of what he himself used to be.

About his own family, he kept quiet, until a few years later, when he was living in Utah, and married to his third wife, Dawne, whom he met on the internet. He had a new laptop. He had never concealed his real name and one night up popped the message: "Looking for Bobby." I thought, it's probably a nut job. So I wrote, 'Where were you born? What's your middle name?' David responded, 'Shut up. I'm your brother.' Oh, my God." Bobby makes a gesture with his hands, clutching his stomach. "It was the most emotional thing I ever felt, the most dramatic event of my life. I thought this can't be happening. I had given up mentally ever seeing my family again. From the whole of my being ..." His voice cracks. "It was like a floodgate opened."

David rang Alfie. "I said, 'I've got him.' 'Nah, you haven't.'"

It had been arranged that Bobby would ring on Alfie's landline. "It was 4pm our time on a Sunday afternoon," Alfie, a details man, remembers. "We have our dinner about 2.30/3pm. David was round. Wendy. My sons. We had had a lovely dinner and just got down. I was still a bit doubtful. The phone rang. I picked it up. I said, 'Bobby?' And ..." Alfie croaks, he can't carry on.

"It was very emotional, like an explosion, a bomb going off," David says.

"I'm still feeling it now," Alfie says.

David rolls his eyes: "All right. You've won the Oscar."

Alfie gets out some kitchen roll from his pocket and wipes his eyes: "As soon as we started talking, we couldn't talk – could we, Bob?"

Bobby shakes his head. Tears are pouring down his cheeks now too.

He couldn't wait to get to England. His daughter Tracy, the baby he had walked away from on the Isle of Wight all those years ago, was waiting for him at the airport. She had been told by her mother that he was buried under the Westway – the A40. Bobby couldn't take his eyes off her in the car. "Looked a bit like Mum," he tells his brothers.

The boys met at the Holiday Inn in Holborn, central London. "I thought they were entitled to kill me. Not literally, psychologically. I thought I was opening a can of worms. But Alfie said, 'Bobby, get here as fast as you can. You will be in seventh heaven. We are going to love you to death.'"

There were questions, the past to unravel, police files to unearth, but that came later. "It was just the thought he was alive!" David says. "We was all crying and kissing. We sat there after all those years apart and looked at each other, thinking, is it real?"

Now Bobby sits with both hands clasped, almost like a penitent.

Is he forgiven?

"Yes," David says, "even if he is one of them petrol tanks. Yanks."

Alfie shakes his head. "Never held anything against him," he says.

• Bringing Down the Krays by Bobby Teale is published by Ebury Press for £14.99. To order a copy for £11.99, including free UK p&p, go to theguardian.com/bookshop or call 0330 333 6846