Alan Bennett, now 81, is as dapper as you like in knitted tie and red-soled grey suede shoes. He still has that schoolboy mop of blond hair, and were it not for a little stiffness as he rises from his chair, and deep veins on his hands, you would take him for a much younger man. We are talking in his front room in Primrose Hill in London. His partner of 23 years, Rupert Thomas, is the editor of the World Of Interiors magazine and it shows – the room is a comfortable cave of 18th-century pictures, a mantelpiece loaded up with cards (one handmade in the shape of a red-soled grey shoe) and a wall lined with books: fat Pevsner architectural guides, the journals of Anthony Powell, Virginia Woolf and John Cheever, Claire Tomalin’s Thomas Hardy biography, and volumes on the history of blue-and-white china.

In Bennett’s plays, bookshelves often loom up to intimidate or overwhelm characters. They are also useful decrypters of taste, and this famous literary diarist is clearly a connoisseur of other literary diarists. (The tone of his journals, excerpts from which are printed in the London Review Of Books each January, is nicely encapsulated by the headline given to the 2010 batch: “Alan Bennett eats a poached egg.”)

What dates Bennett is not his appearance or indeed his bookshelves, but his grasp of technology. His new film, The Lady In The Van, is about the extraordinary figure of Miss Shepherd, who for 15 years, from 1974-89, made her home in a van parked in the driveway of the house where he used to live, a few streets from his current home in Camden Town. Bennett himself is played by Alex Jennings, who is seen at various points stabbing ineffectually with two fingers at a manual typewriter. This, I assume, is a little cruel. “No, no, it’s absolutely true,” Bennett says, in that blissfully mournful voice that could have been designed for voicing Eeyore (which, of course, he has done, reading the Winnie the Pooh books for BBC radio). The action of the film takes place in the 1970s and 80s, so I assume that he has long since moved on into the digital world. “Well, we had a computer, but we were burgled and that was the one thing that was taken,” he says. “I was relieved, really. There was nothing on it. I didn’t know how to put work on it.”

Surely he at least uses an electric typewriter? “Well, electric typewriters, they hum. They are waiting, you see, for the next note.” So an AEG Traveller de Luxe manual typewriter it is – lacking both expectant hum and flickering cursor. As we speak, a memory about Bennett surfaces, something that Thomas told me when, nearly 20 years ago, he and I worked together on the World Of Interiors. Is it really true, I ask, that you keep your work-in-progress in the fridge for safekeeping? He giggles. “I used to. But then a friend in New York said, that’s the first thing that goes up – it’s the real hotspot in a domestic fire, and you’d be roasting your manuscripts.” He used to know he was getting somewhere with a project when the urge took him to pop his pages in the salad crisper. His old manuscripts (now mostly in the Bodleian Library in Oxford) are stained, he says, by various unnamed drips and spillages: “If anybody’s foolish enough to go through them, they’ll be able to see.”

The Lady In The Van began as a series of diary entries. Assembled into a single narrative, they were first published in the LRB after Miss Shepherd’s death. Later he turned the story into a stage play, starring Maggie Smith, and now comes the film, directed by Nicholas Hytner, also starring Smith. Jennings does a lovely version of the writer, a role he first took in 2012 in another Bennett play, Cocktail Sticks, at the National Theatre. “My feeling is that probably he’d been doing imitations of me for years and when he did Cocktail Sticks he was simply going public,” Bennett says. In the film the author is trapped between the demands of two elderly women: his mother, declining up in Yorkshire, and Miss Shepherd, whose unruly presence tests the liberal principles of the new generation of artists, television people and journalists who, Bennett among them, had moved into Camden Town and done up the big old Victorian villas there (the “knockers-through”, Bennett has called them).

I tell him I still don’t understand why Miss Shepherd ended up in his driveway; for years, visitors to the house were obliged to sidle past her parked van. “It wasn’t, at the time, a humanitarian gesture,” Bennett says. The van had been parked at various points along the road, and at length “drifted down” to his house. “I worked at the front window. There were so many incidents when she was not actually physically roughed up, but people banged on the side of the van, hoping to get her out and waving her stick, that I always had one eye on her and it stopped me working. Bringing her into the drive was a way of getting rid of that and enabling me to find a way to work. In retrospect, it might seem as though it was humanitarian, but the larger part was pure selfishness.”

He says it’s exactly the same with a robin that nested this spring in the garden. “It raised two fledglings: one fell out of the nest and was killed. Then there was the other one, which came out of the nest too soon and had to be fed by the robin. I was writing overlooking the garden and it was exactly as it was with Miss Shepherd: half my eye was on this bird because there were jays about, and seagulls. If I could have taken the bird inside, I would, not because of the bird but because of me. I’m sure that’s not goodness, really.”

In the film, there are two Alan Bennetts. One sits at his desk, pen poised, and issues barbed remarks to the other Bennett, who flutters about timidly, not knowing quite what to do. The Bennett at the desk knows, on some level, that Miss Shepherd may well end up providing him with “good material” (though, he points out, “I wasn’t nurturing her as a possible subject”). This sense of doubleness runs through Bennett’s writing: in the diaries, there is often a part of him taking critical notice of what he is (or isn’t) doing, a kind of extreme self-awareness. For example this, from an 80s fly-on-the-wall documentary he collaborated on, about a hotel in Harrogate: “For years, hotels and restaurants were for me theatres of humiliation and the business of eating in public every bit as fraught with risk and shame as taking one’s clothes off.” The performance of going to a cafe would include his parents ordering a pot of tea, then smuggling bits of bread and butter, brought from home, to the young Bennett and his brother under the table while the waitress wasn’t looking – enough to turn any boy self-conscious. Bennett’s work is full of timid characters overwhelmed by and attracted to bolder, flashier types: Kenneth Halliwell and Joe Orton in the film Prick Up Your Ears, for example; Benjamin Britten and WH Auden in the play The Habit Of Art; even Bennett and Miss Shepherd in The Lady In The Van.

Bennett’s life has been scattered with larger-than-life types, such as Russell Harty, the broadcaster who died in 1988. Their Yorkshire childhoods were similar but Harty had, according to Bennett’s address at his memorial service, “learned ... by the age of 20 a lesson it took me half a lifetime to learn, namely that there was nothing that could not be said and no one to whom one could not say it”. (As a student, Harty invited Vivien Leigh round to his rooms for drinks, a most unBennett-like act.) Bennett portrays his fellow performers in Beyond The Fringe (Peter Cook, Jonathan Miller and Dudley Moore, with whom he made his breakthrough on Broadway in 1962) as larger, more vivid figures; he, in his own accounts, tends to be waiting in the wings nervously while the audience laughs itself silly at Cook’s jokes.

“My parents were quite shy and, looking back, it felt they had made ‘shy’ into a virtue and I believed that,” he tells me. “Then as I got older I realised it’s an affliction, really, and if you’re shy you are just as self-centred as anyone who is outgoing and the life and soul of the party.” But perhaps also more watchful; the one more likely to be making pungent little notes in the day’s journal, or transforming his close observations into works of art such as the Talking Heads monologues of the 1980s and 90s, or The Madness Of George III.

I visit on the day Jeremy Corbyn gives his first speech as Labour leader to the party conference; I ask Bennett if he caught any of it. No, but he is a strong supporter. “I approve of him, if only because it brings Labour back to what they ought to be thinking about. The notion that they were wanting to choose somebody who would win the election when we are three, four years away from one, regardless of what half the nation wants – that seems to me far more risky than what they have done. I’m sure Corbyn is a risk, but I very much approve of him.”

Bennett reckons the opposition leader is “absolutely right” about nationalising the railways. He and Thomas use the east coast mainline regularly to visit their house in Yorkshire; the line was renationalised after the financial crisis in 2009, but privatised again early in 2015, which he says “was pure ideology” on the part of the Conservative-led coalition. “It was making a profit and there was no reason to sell it back again. No good arguments have been advanced for it. And obviously the National Health and so forth: everything Corbyn’s said seems to me to ring bells. The other candidates had stopped talking about these things.”

The Conservatives, meanwhile, are “quite close to a totalitarian attitude”, he says. “It’s not merely that they want to be the governing party, but the only party, and that’s never been part of the British political tradition. That stems originally from Mrs Thatcher: she did believe that Labour was wicked.” In an address he gave last year at King’s College, Cambridge, Bennett raged (quietly, but it was rage none the less) against private education, suggesting that it ought to be phased out: “Private education is not fair… Those who provide it know it. Those who pay for it know it. Those who have to sacrifice in order to purchase it know it. And those who receive it know it, or should.” In the same sermon, he wrote: “There has been so little that has happened to England since the 1980s that I have been happy about or felt able to endorse. One has only had to stand still to become a radical.”

He is a royalist as well as a socialist. In Bennett’s very funny story The Uncommon Reader, the Queen discovers a taste for novels – and thereby an inner existence. “The Queen comes upon herself, comes upon having a private life that she has never really thought about,” he explains now. “She’s a joy to write about, because you don’t have to do very much. Everybody knows how she speaks, what her circumstances are, so you’ve only got to do a tiny bit. She’s a gift.” In his 1988 play A Question Of Attribution, about the art historian and spy Anthony Blunt, the Queen was played on stage by Prunella Scales. “I think Prunella Scales got her precisely right on the stage. The glamorising of her that has happened has nothing to do with anything, really – it’s just exploiting her.” I assume he means Helen Mirren’s more recent take – in the film The Queen, and on stage in The Audience. “I don’t mean Helen Mirren herself isn’t good – it’s just that it’s such a construction, really.”

Mirren was a good egg, he says, last time he performed on stage. For the National’s 50th anniversary gala in 2013, he performed an excerpt from The History Boys, in which he took on the part of the wayward teacher Hector, originally played by the late Richard Griffiths. “I was so nervous it was painful, really. They did it twice and the first time I slightly messed it up – though since the lines were in French hardly anyone knew. But I knew. And then the next night I did it properly and it was all right, but oh, I wouldn’t want to do it again. And when I came off that first time, Helen Mirren said, ‘Oh, fuck ’em, fuck ’em.’ Which is really all you want, someone to say that.”

I wonder what he makes of the fuss surrounding Corbyn’s not singing the national anthem at the Battle of Britain memorial service in September. “Oh,” he says, “that was the Daily Mail. The lies on the front page of the Mail are so vulgar and glaring. Occasionally people say they like my work and then I see they have a copy of the Mail, and you think, ‘Well, how can you?’” The irony is, he says, that “half the royal family don’t even sing the anthem… they don’t even seem to know the words to Jerusalem. The notion that you are required to sing the national anthem in order to prove your patriotism, and if you don’t you’re not patriotic, is so absurd.”

Like all men of his generation, Bennett did national service. He was sent to train with an infantry platoon in Pontefract that was going to be sent to the war in Korea; not unreasonably, he didn’t want to go. He worried because he had made the mistake of being an extremely good shot – not with a rifle (“I would shut my eyes because of the recoil”) but with a Bren, a light machine gun.

Everyone at his school knew there was a possible get-out clause that could be invoked by bright boys: the Russian course, an intensive language training aimed at providing interpreters, translators and sometimes intelligence officers. Writers Michael Frayn and Dennis Potter did it, as did the late John Drummond, controller of the BBC Proms in the 90s, who ended up working in intelligence; Bennett applied and got on.

I had harboured a small and ludicrous fantasy that Bennett might also have ended up spying, but am swiftly disabused. “They would never have asked me to be a spy – I am the wrong class, you see,” he says. The course, though, was “lovely, idyllic”. He had never been away from home before (he was raised in Leeds, except for a brief period when his parents moved to Guildford). And there he was, in Cambridge, and with a bit of money, too. Having already been offered a place to study at the university the following year, he realised that other people on the Russian course had won scholarships – so he reapplied, this time to Oxford. Neglecting his Russian in favour of preparing for the scholarship exam (in history), he ended up “busted back to private”, on standard national service duties. It was the mindless, repetitive and demeaning nature of that work that put “the iron into my soul,” he says. “I became more leftwing. I was quite conservative and Christian and rather priggish when I went into the army. That began to go, I hope.”

Bennett has written that he gave Guy Burgess some of his own sentiments in his one-act play and film about the spy, An Englishman Abroad – lines such as: “I can say I love London. I can say I love England. I can’t say I love my country, because I don’t know what that means.” Does he think patriotism is a necessary virtue? “Well, I don’t really,” he says. “To be patriotic – to be English – is partly to be sceptical of one’s country, and of patriotism itself. That is true of me, I think. Burgess and Blunt wanted a kind of moral solitude, which is a very different thing. They didn’t want to be beholden to country. Being sceptical about patriotism is a part of patriotism – a refined sense of patriotism, I think. Demonstrations of patriotism always make me uncomfortable, but I don’t think that makes me less of an Englishman.” He laughs. “I don’t know. It makes you sound so pompous if you put it like that.”

While Bennett’s audiences and readers have come to know all kinds of apparently intimate things about him – the pigskin suitcases his Aunty Myra brought back from India during the war, the particular smell of his grandmother’s dresser – he has kept much of himself back. Despite the sense of familiarity that comes from having watched his television plays of the 70s and 80s, or read his autobiographical stories, he is not in any real sense knowable. Or rather, he is easily mistaken: those wry Yorkshire tones and an art that tends to dwell on matters domestic and provincial mean he is frequently regarded as rather cosy – when, in fact, plays such as The History Boys and People only thinly disguise his political anger.

When I ask him about this gap between public and private, he talks, unprompted, about his sexuality. “It’s a thing about being beholden, about not being beholden to anyone,” he says. “I didn’t want to be in anybody’s pocket, that’s why I didn’t want to be thought to be gay, particularly. Pigeonholed. And then as you get older it just ceases to matter. The times changed and then... Rupert and I have been together for 23 years, you see, so it’s never been an issue since then.”

He wouldn’t have come out while his parents were alive, anyway. “It would have distressed both of them. People talk about coming out to their parents, and I just didn’t see the point of that. In my case, coming out about what? Nothing was happening – I might be gay but I didn’t have a partner or anything, so there was no point.” There was also, anyway, a decade-long relationship with a woman, Anne Davies, who died in 2009. “It started off just as a fling, really, but we became very fond of each other. She overlapped with Rupert and so it ended, and then she came to Yorkshire and lived next door. To begin with, not surprisingly, she didn’t get on with Rupert, but then she became ill and she became closer to Rupert than she was to me, really.” She was, he says, “very beautiful”. At the time of the relationship he enjoyed the sense of wrongfooting people: “I think I probably got some satisfaction from thinking, ‘Well, everyone assumes I am gay.’”

These days he doesn’t go out all that much, certainly not to the theatre. “I used to go to the National when Nick [Hytner] was the boss because I could go in and sit in the producer’s box and watch an act, and then you could go home, if you wanted. I don’t like being imprisoned in an audience now, partly because I often want to go to the loo.” Hytner throws two big parties a year, one in the summer and one in winter, which Bennett enjoys, preferring theatre people to literary people, and enjoying catching up with the actors he has worked with. “It’s a much kinder world, is the theatre, than literature. It seems to me that they are more forgiving of one another.” Either through loyalty or innate conservatism or a bit of both, Hytner has directed all Bennett’s recent plays and films, and his professional relationships are exceptionally enduring: his diaries and essays are almost invariably published in the LRB, edited by Mary-Kay Wilmers, a former neighbour in Camden Town, another of the knockers-through.

“Nick chose me, rather than the other way round,” he says of Hytner. The director approached him in 1989 to write an adaptation of The Wind In The Willows for the National Theatre, then run by Richard Eyre. Their first meeting was in a restaurant in Primrose Hill, and Bennett spent much of the dinner explaining the area to him. The intensely contained Hytner never owned up to living streets away himself. Professionally, Hytner “gets the mixture of encouragement and criticism exactly right,” Bennett says. “So that when you show him something, he is enthusiastic about it but he nevertheless has reservations. Both those things make you want to go on with it – both the praise and the criticism. I don’t know whether he does it instinctively or whether he does it with other people, but it always works for me.” It used to be like that at the BBC: Bennett worked for 20 years with the late producer Innes Lloyd, on dramas such as A Day Out (his first play for television, broadcast in 1972) and Talking Heads. “I wanted to please him in the way that I want to please Nick. Innes was exactly what was good about the BBC. You told him what you wanted to do, and he would arrange things without letting you know there was any problem at all. The other producer like that, in a different way, was Ned Sherrin: he understood about giving you space and not letting you know there was a row on. I’ve never really been able to do much at the BBC since.”

He watches telly: he enjoyed the recent adaptations of Cider With Rosie and An Inspector Calls. The latter is one of two JB Priestley plays he would like to have written himself, he says, the other being When We Are Married. He loves the comedy of Stewart Lee, and the geeky sitcom The Big Bang Theory, and the “funny but so savage” playwright Martin McDonagh. He got into trouble a few years back when he said in an interview that he didn’t read contemporary British fiction, which “was taken as being an affront to AS Byatt and everybody else – but I just read other things”. Recently, it’s been The Shepherd’s Life, a memoir of farming in the Lake District by James Rebanks (“a very good book”). But he is also reading some new British fiction – Grief Is The Thing With Feathers, Max Porter’s debut novella, inspired by Ted Hughes’ Crow. “More poetry than anything else but it’s good,” he says. Bennett is more of a Larkin than a Hughes man, finding Hughes’ work “a bit Lawrentian” and the man himself, when he met him, “condescending. He obviously thought I was a bit showbiz and corrupt”. Of Hughes’ and Plath’s poetry, he says: “I slightly felt with the pair of them that if anyone had cracked a joke the whole thing would have split wide open. The whole thing was just so ‘poetic’, in inverted commas. Larkin is about things I recognise. I see that poems like [Hughes’] The Pike are wonderful, but they aren’t about the life I lead.”

His own work continues “slowly”: he goes to his desk each morning at 10.30am; and Hytner has an old screenplay of his, once written for John Schlesinger but never made, that might turn into something. Things are somewhat becalmed as Hytner, having left the National, has not yet started up his new venture, a London theatre near Tower Bridge, planned to open in 2017.

Sources of domestic tension, meantime, are few. “I’m one for being early, and Rupert’s one for being last-minute. It’s never settled. If we were ever to split up it would be about that, really. I’m chary of saying to people that I’m very happy or even that I’m happy. You tell people that you’re happy and they are bored by that. But we are.”

• The Lady In The Van is released on 13 November