Charlie and the Chocolate Factory has taken five years to become a stage musical, for reasons logistical – Sam Mendes, the director, was out for three of them doing Skyfall – and practical: the book is a tricky one to adapt. There are the kids; the old folks in bed; the pyrotechnics of the chocolate factory. There is the ambiguous character of Willy Wonka himself. And there is the question that hangs over the entire production: what on earth to do about the Oompa-Loompas. "It's big," Mendes says of the task before him. "Christ, it's so big."

We are in a rehearsal space in south London, where the company is going through its paces before moving to Drury Lane. Anticipation for the show is feverish, thanks to the success of Matilda, another Dahl adaptation, and Mendes's post-Bond nuclear glow. The Dahl estate spent years rejecting offers for other stage adaptations. They said yes to Mendes for sound reason: Neal Street Productions, his company with Caro Newling and Pippa Harris, has staged some of the best musicals of the past decade – most notably the brilliant revival of Sunday in the Park with George. If Mendes pulls this off, he will be loved for a while and forgiven his other successes.

And so here he is, surrounded by a fascinating team: Scott Wittman and Marc Shaiman, the American songwriters, are accompanied by a largely British core of writers, choreographers and designers, some plucked from relative obscurity in a golden ticket-like scenario. David Greig, who is adapting the book for the stage, is a Scottish playwright from a wholly different background from the Broadway duo, who, when they heard they'd be working with him, looked up his plays. "They're all about terrorism," Wittman says with some bafflement.

Greig was equally bemused. He has a great track record with the Royal Court and the RSC – he put Alan Cumming in a production of Euripides' The Bacchae, at the Lyric – but had never done a musical. He thinks he got the job, he says, because at their first meeting, rather than doing jazz hands for Mendes and hustling for the job, he said with gloomy integrity, "I think it's really difficult. I think it's nearly impossible."

There are lots of problems with the book-to-stage conversion of Charlie and the Chocolate Factory, which, unlike other Dahl stories, doesn't follow a conventional narrative. Charlie Bucket starts out a nice boy, and finishes up a nice boy; there is no progression. There is also no baddie – or, rather, the baddie is to some extent Wonka himself. It's never quite clear how much danger the children are in on their factory tour, but there is an implication, Newling says, that "he sends these children to their deaths. He tests them." This is a large part of the appeal, of course: "People love the deliberate wickedness, but it cuts to the core of what's right and wrong; what's fair," Newling says. It is a story about good and bad parenting.

It is also a story about the eccentricity of genius. "There's no question that there's a Willy/Roald Dahl crossover," Mendes says. "When you listen to Dahl reading the book on the audio tape, he's frightening. He's not some avuncular, twinkly old professorial leprechaun. He's all sorts of things." An epic git, is obviously the current read on Dahl, which has done nothing to undermine his popularity. Nor should it: Dahl's gittishness is at the core of his professional power, that fabulous spikiness that militates against the tweeness of so many kids' productions. "That is the key in many ways," Mendes says. "And that for me was one of the big attractions of doing this. I don't feel that the Willy I perceived in the book had been rendered yet in either of the films, however wonderful Gene Wilder and Johnny Depp are."

Wonka will be played by Douglas Hodge, a casting decision that probably couldn't have been made on Broadway. Hodge's background is classical – he has done a lot of Pinter and was in Titus Andronicus at the Globe and Pericles at the National, which will give Wonka a cultural heft, half pop, half canon. There is no Wonka back story, so Hodge is operating on an assumption of timelessness, he says. "I think of David Bowie, or Michael Jackson – those one-off characters who've been rather separate from the rest of the world for a long time and who have a weird innocence."

This is the first time Wonka has been played by a non-American, and Hodge drew on traditions of Gilbert and Sullivan, Noël Coward and Rex Harrison for the role. "Although Gene Wilder's gentle, transcendental, slightly druggy Wonka was marvellous, I didn't quite understand the Johnny Depp-channelling-Anna-Wintour version. This is very different. I'm enjoying the Englishness." Ticklish and giggly is the tone he is after, but also, at the outset, disillusioned. "I think, at the beginning, he's lost faith in innocence. I think it might be darker than it's been played before."

If there is a cure for withstanding the pressure of filling 2,200 seats at Drury Lane every night, of reinterpreting a book beloved by millions, and of handling relations between Warner Bros, the show's co-producers, and the finicky Dahl estate, it would be directing a Bond movie immediately before.

Mendes is back on home ground. In the 1980s, he directed his first plays while studying English at Cambridge, and fresh out of college landed a gig directing Judi Dench in a West End production of The Cherry Orchard. (She wrote about it in her memoirs; what an upstart he was.)

After that, Mendes joined the RSC and put out productions of The Tempest, Richard III and Troilus and Cressida. His greatest productions have been in musical theatre. Musicals date fast, but when they work, he says, they are "the best and the purest form of theatre", and he has a brilliant track record; his 1993 revival of Cabaret is one of the best things I've ever seen on stage.

The film record is more divisive: American Beauty, Mendes's first movie, won the best picture Oscar in 1999 and was variously loved and loathed. He garnered six more Oscar nominations with Road to Perdition and made a good adaptation of the Richard Yates novel Revolutionary Road with Kate Winslet, his ex-wife.

Bond is his biggest film, and the most successful in the franchise to date – it has grossed $1bn worldwide. "The biggest movie I've ever done," Mendes says, "followed by the biggest show I've ever done, almost back to back – not a sensible piece of planning."

Nor, to some of his peers in Hollywood, a sensible career trajectory. Mendes's habit of turning down movies to labour on London theatre projects that only a relatively tiny number of people will see is eccentric to the point of deranged in LA, but he finds it psychologically necessary. He was offered another Bond film almost immediately after this one, but wasn't tempted. "Was I willing to go straight back into a room with a writer and start work on the same set of characters and the same scenarios as I've been working on for the last three years? No. The idea made me feel physically ill." He laughs. "The movie world does not acknowledge the theatre world as a serious entity, on the whole. So you get a lot of, 'Why would you do another play?' "

You also get, "What do you say to all the Bond fans you've disappointed?", which makes Mendes laugh. "I say to them, my life is not a democracy. It's not up for discussion. What? Maybe I should go on to one of the [Bond] chat threads and change my mind? I don't think that's going to happen."

If there is a crossover between Bond and Charlie, you could say it revolves around the topic of Englishness. (In the opening scenes of the Bucket family's shack, Mark Thompson's set design for the stage musical has a Lowryesque air to it, appropriate for a recession-era musical.) Making Skyfall, Mendes thought he was being esoteric with the themes: "Under the surface of the movie is a meditation on ageing, and loss, and England. And what it is to be English, and does it mean anything now?" It is the story, as he puts it, "of someone who disappears for a while, and comes back to England to find everything has changed, but everything is basically the same. And that was basically what I was going through."

After the end of his marriage to Winslet, Mendes moved back to London from New York. (The split, by all accounts, was amicable, and he doesn't miss being part of a megawatt celebrity couple. "I have immense admiration for Kate – how she coped with it, and still does," he says. "Better than I did. I didn't like it." He is now with Rebecca Hall.)

Anyway, the shift back to England was a welcome one. "I tried a bit to become more American when I was doing movies, but I never penetrated certain key things. I don't know how to talk with my crew about what happened last night in the sport. Whereas here I do. I don't know how to read a newspaper in America properly, whereas here I do. I don't know how to watch the news from beginning to end in America, but here I seem to manage to. I like the smell of the streets in the rain here, whereas there I would always want to go inside. This is my home. I never felt it was a permanent thing, being over there. I love New York, but I'm English, and I still feel English and I did when I was there."

His and Winslett's nine-year-old son, Joe, has been around during Charlie rehearsals and his father road-tests some of the material on him. But in any case, Mendes says, the key to a production like this is "locating your inner child, trying to remind yourself of what it was like when you first heard the story. And you have to unremember the narrative; in that respect, it's a little like doing Bond, where everyone in the audience knows that Bond's not going to die. But you have to make people believe that there's a danger he will."

So he does the next best thing – I'm assuming everyone has seen this film already – and kills off Dame Judi?

"Ha! You can't kill him; let's kill her instead! I thought I was going to get so much shit for that. But you know you shock people into rediscovering their first acquaintance with the characters, and it's a similar game being played here. Which is to try to make it impossible for Charlie to win, and then watch him win."

What was it Dench wrote about him in her memoir? "I paraphrase, but she wrote that when I was doing The Cherry Orchard, she said to me, 'Could I try this?' And I said, you can try it, but it won't work. And she thought I was – entirely correctly – completely jumped up."

Was he?

"Oh, I was a pain in the arse. Appalling. Obnoxious and all the things you would expect from a 24-year-old directing Judi Dench in The Cherry Orchard. What the hell did I know? I shudder to think of it now. And then, of course, when we did Skyfall, I said to her, 'Could you try this?' And she said, 'I could, but it won't work.' She said, 'I waited 25 years to get back at you…' "

It is corny, Mendes says, but being back in London and working on Charlie reminds him of what makes him happiest in this world: the deep, satisfying security of working with people he has known for most of his career. The first rehearsal for Charlie was in a church hall in Brixton and he was in a horrible mood that morning, grousing at how far it was, exhausted from Bond, and then walking in and seeing all his old friends.

"I thought, 'Oh yeah, I remember this. This is great. Oh yes, it feels like coming home.' I've never felt that on a film set. Film sets don't have that sense of security." (Because of the success of Bond, some of the actors auditioning for Mendes experienced higher than average anxiety. "Yes, they were very nervous. You try to put them at their ease, but I'm not a particularly fierce presence in the rehearsal room. I don't think I'm scary for long.")

The chances of Charlie being good are extremely high, with that combination of posh/brainy/commercial that Neal Street does so well and in the face of testing logistics. The idea of doing a chocolate river was eventually abandoned, but there are other pyrotechnics, and although the show isn't selling itself on gimmickry, the amount of weight hanging over the stage called for steel reinforcement. From a design point of view, Mark Thompson says, he hopes this production will correct earlier versions of the story. "I didn't like the [recent] film," he says. "There was lots of wonderful things in it, but I felt overall it was quite chilly. Dare I say it, an ever so slightly cynical view of the world? And I thought the first film was charming, but a little antiquated."

If the show succeeds, a large part of it will be down to the choreographer, Peter Darling, who was responsible for Billy Elliot and Matilda, and who works on the principle that children should move like children and not show ponies. "You have to understand the parameters," he says. "If you're trying to make a child move in a very expressive, slow, lyrical way, that's not going to happen. You could waste days on that, because children, by the age of 10, have only just come to grips with balance and coordination." Instead, he opts for "lots of sharp, jerky, off-centre movements, and that suits me quite well, because that's probably how I move".

The golden ticket-winning children in Dahl's story will, in part, be characterised through Darling's choreography, so the three boys playing Mike Teavee were chosen for their "kinetic energy" and will move in something like street dance. Augustus Gloop? "He expresses himself through eating, so the boys have learned some magic tricks around eating. There's a degree of embarrassment in him, so his inertness is what makes him unique."

Neither Charlie nor his co-stars were brought in from the dance schools: they are regular kids, and, Darling says, "there's something beautiful about watching children who have never danced, and who think they can't dance, actually find they can. It's magical."

"It's like being a schoolteacher," Douglas Hodge says. "At one point, there are 17 children with every germ known to man in the room. They're so good and so fearless. It's rather alarming, and heartbreaking, to watch them."

And so to the problem of the Oompa-Loompas, described in Dahl's first draft as "pygmies", which even he realised was a bit dodgy, and presented in the final book as a tribe of jungle dwellers "saved" by Wonka and shipped in a crate to his factory, where their love of chocolate inspired them to work in contravention of EU labour laws. There are multiple questions: "Are they real?" David Greig asks. "And if they're real – what? Willy Wonka has an army of tiny slaves? Is that what we're saying?"

They are one of the big reveals of the show, so no one will tell me what they're going to look like, although it involves an element of puppetry (created by the brilliant Jamie Harrison, who made the illusion of Tinkerbell as a ball of fire for the National Theatre of Scotland's production of Peter Pan.) Some of the Oompa-Loompa costumes are hanging at the back of the rehearsal room and, unoccupied, look like Teletubbies on a clothes line.

It's impossible to imagine how the final thing will look, but, given the people involved and the many years of preparation, it's no surprise there's so much buzz about it. With expert hype deflation, Mendes says you can't let these things affect you. "You can theorise all you like," he says. "You can go on and on about it and try to dress it up." He smiles. "But, ultimately, it's just a story told with joy."

• Charlie And The Chocolate Factory previews from 17 May at the Theatre Royal, Drury Lane, London WC2; go to charlieandthechocolatefactory.com for details.