The intimidating front door could belong to a gothic castle. I half-expect it to be opened by a giant butler called Lurch with a forbidding: “You rang?” Instead Jaqueline Wilson, aged 77 and slight as a pipe cleaner, answers with a smile warm enough to heat her huge home. She leads us into the living room, and before we know it there are drinks, doughnuts and chocolate biscuits in front of us. I’m here with my younger daughter, Maya. There was no way Maya was missing this. She is one of many young people whose lives were transformed by Wilson. Maya was a late reader. Then she discovered Wilson’s novels about kids struggling to find their place in the world and was hooked.

That was in the early 00s, around the time Wilson published Sleepovers, one of her most popular novels. Twenty-two years on, she’s finally written a follow-up, The Best Sleepover In the World.

Wilson had considered Sleepovers one of her minor works – it’s a short book for readers aged seven and over. But the more she asked fans which of her books were their favourites, the more she realised how popular it was. Wilson has written well over 100 books, sold about 40m copies in the UK alone, been translated into 34 languages and was for years the most borrowed author from British libraries. Sleepovers is one of her five best bestselling books, and she believes there are two reasons. First, so many children love sleepovers. And second, it’s about something at the heart of young (and older) lives – friendship and friendship betrayed.

“Children not having a best friend or losing their best friend is a really serious thing,” she says. “Parents often take the attitude: ‘Don’t worry – you’ll get in another group soon,’ but the agony of going around the playground by yourself is awful. I also wanted to show you’re not alone if that happens to you. There’s nothing wrong with you – it’s just the way life works out.”

It’s why Wilson’s books resonate with so many young people, especially girls – she understands how they feel; how tough it can be to grow up. Early on, her books were dismissed as being about and for outsiders. What Wilson understood from the off is that lots of us think of ourselves as outsiders.

So much has changed since the largely analogue days of Sleepovers. Not just in terms of technology, but attitudes. In the first book, Lily, the non-verbal, wheelchair-using sister of the narrator Daisy, is largely passive. In the new book, Lily is the hero – sassy, communicative (she communicates using the sign language Makaton) with a super-cool disabled best friend. The girls are the same age, but the setting is bang up to date. Now when the snotty bully Chloe is desperate to show off in front of her peer group, she does so by boasting that a TikTok influencer will be coming to her sleepover. It turns out the author has strong views on TikTok, and the digital world in general.

Wilson’s life has also changed dramatically since Sleepovers was published. She has been awarded an OBE, been made a dame, has served as children’s laureate, suffered heart failure and had a kidney replacement. Back in 2001, she had recently separated from her husband of 30 years and was single. For the past 21 years she has been in a relationship with her civil partner, Trish, who is here today. Seven years ago, they upped sticks and moved from Kingston upon Thames in south-west London to the Sussex countryside.

“A lot of friends thought I would hate it and would want to come back immediately,” Wilson says. “What’s so lovely now is that we’ve made more best friends here than we’ve ever had,” she says. She adds that she’s never been so happy.

“You’re not the townie that people thought you were, are you?” Trish says lovingly.

This rural idyll is a far cry from her childhood. She grew up on a council estate in Kingston, the only child of parents in a loveless relationship. Her mother was a “terrible snob” who insisted that theirs was a better class of council estate and that she’d only ended up with Wilson’s father because all the good men were away fighting in the war. Meanwhile, he had a terrible temper. “He’d give you the silent treatment for weeks. It was quite scary. I think they were both very unhappy people.”

Wilson says she was a disappointment to her mother. “My mum would have loved Shirley Temple as a daughter – full of confidence, tap-dancing all over the place in flouncy clothes, and showing off.” And what did she get? “A girl sitting there reading a book, looking gormless.” Her mother chose Wilson’s clothes till she left home and refused to let her wear jewellery. She points to the huge rose quartz ring on her finger, marking her civil partnership with Trish, and grins. “I mean, isn’t it pathetic when, even in your 70s, you wear things that a psychiatrist would point out is rebelling against your mother?”

You only need to listen to Wilson for a few minutes to understand how much her childhood has shaped her stories. So many of her characters are misunderstood, disadvantaged and vulnerable despite a tough veneer. Tracy Beaker is a classic example – she has a feckless mother, lives in care and is desperate for love. Yet on the outside, she’s a toughie who bloodies the noses of those who disrespect her and tells us repeatedly that she never cries (though she does admit to getting severe hay fever when upset). “I didn’t have as harrowing a childhood as some of the characters in my books,” Wilson says. But it was no picnic.

Books were Wilson’s escape. First reading them, then writing them. She wrote a 21-page “novel” (Meet the Maggots) at nine, a “rambly” full-length novel at 15, sent her first to a publisher at 19 (“Not quite for us, but we liked the characterisation,” they replied) and had her first book (Ricky’s Birthday) published at 23. By then she’d already had a full life – leaving home at 17 to write for the girls’ magazine Jackie in Dundee, marrying at 19 (another dysfunctional relationship) and having her “wonderful” daughter Emma (a professor at Cambridge university) at 21.

Her books went largely unnoticed for many years. One editor told her: “Remember, Jacqueline: people don’t like reading about little people.” She made a decent living writing true-life stories for confession magazines, but she never expected to live in the luxury she does now.

The living room and hall are crammed with all-sorts: artworks galore, ceramics, rubber toys (for her two dogs, Molly and Jackson), a telescope, a rocking horse, and on it goes. Dominating the living room is a huge, immaculately ordered bookshelf. There are children’s books, adult books, first editions wrapped in protective plastic, autographed books, research books on the dietary habits of foundlings. Even the table in front of me turns out to be a book – underneath the glass top are the loose pages from an ancient copy of Jane Eyre that has fallen to bits.

Who’s more into books, I ask.

“Both of us,” Wilson replies. “I’m more extravagant. Since we’ve moved here, Trish has bought more nature books.”

“Mainly books on meadows,” Trish says. “Not so many novels these days.”

How many books do you own between you?

“I am really hopeless about numbers,” Wilson says.

“Forty thousand,” Trish replies, instantly.

“Nooooo! Not as many as that,” Wilson says.

“Well it was 35,000 many years ago,” Trish says.

“I would say 15,000,” Wilson says. It’s a rare disagreement.

Wilson and Trish met through books. Trish was a bookshop manager who hosted talks with authors. “I asked her if she’d like to do an event with me. This was in the days when I was doing lots of signings. Terry Pratchett, who was quite picky, said: ‘Have you done one with Trish? She’s really good.’ Trish had met me, but blow me, she’d never asked me to do an event. So I thought: ‘I’ll be bold and ask her.’ And we got on really well together.”

“We didn’t get together immediately,” Trish says.

But it didn’t take long. Wilson, who had never previously had a relationship with a woman, says Trish brought out her competitive streak. “I asked her about earlier girlfriends and she said she’d never had a year-after-year relationship, and I thought: ‘Right, I’m going to be that.’ And I have been, so far.”

Wilson’s success as an author came in her late 40s. It surprised some of those she worked with as much as her. “I heard somebody who worked closely with me say: ‘Fancy little Jackie being so successful,’ and I don’t think she was referring to my height. I think that was a reference to my background and my demeanour.”

She remembers looking at the bestseller list and discovering she was No 7. That also brought out her competitive nature, she says. You wanted to be No 1? “Yes I did,” she admits slightly shamefaced. “But a certain JK Rowling came along and you’re never going to beat that. And there’s always been one or two others much better than me.” Better or bigger? “Bigger!” She laughs. “Occasionally better.”

This was a golden age for children’s literature. Rowling and Wilson endowed so many kids with a love of fiction. Although independent bookshops have recently announced an upturn, according to the Booksellers Association, Wilson worries about the reading habits of the young.

“Children’s books are often to do with something on TikTok, like The Wonky Donkey and Greg the Sausage Roll.

“I’m not knocking these things, but they’re not quite Where the Wild Things Are.” Many bookshops now feature selections based on BookTok, a subcommunity of TikTok.

“What’s missing some of the time is books that require stamina to read. I loved books that kept you going, and if you didn’t understand a word you just learned to get the gist of it and carry on, and if it was vital you looked it up. Now, there is a tendency to make children’s books so easy to read that it makes it hard to imagine most children would go on to read complex wonderful literary novels as they get older. There are a lot of books kids like now because you can read them in an hour.”

Not everything is being dumbed down: “I’m a huge fan of Katherine Rundell, who is one of the most exciting children’s writers ever. But I do think texts nowadays do tend to not be quite as subtle as they used to be and plots are not quite as complex. When you look at children’s classics like Little Women and Treasure Island, it would be odd to find a seven- or eight-year-old reading them happily now. I didn’t come from a posh background, but lots of kids were reading those books in junior school without thinking about it.”

What has changed? “The digital age, mostly. It’s frightening how when travelling or sitting in child-friendly cafes, if a little kid is being fretful or difficult, the tablet is put in front of their face and then they’re watching something.” Wilson says there’s a big difference between reading and watching. “You have to use your brain more when reading. I don’t think parents automatically read aloud to their children now. I know everyone leads very busy lives, but reading aloud to your children when they’re little is one of the best bits of being a parent. You cosy up together, share the magic of the story, and it could be a memory that lasts for ever for your child.”

Is she worried about the increasing censorship of children’s books? A report from the writers’ organisation Pen America said book bans in US public schools increased by 28% in the first half of the 2022-23 academic year. Of the 1,477 books banned, 30% were about race, racism or include characters of colour, while 26% had LGBTQ+ characters or themes. Research carried out by the Chartered Institute of Library and Information Professionals in the UK this year found that a third of librarians had been asked by members of the public to censor or remove books, with the most targeted books involving empire, race and LGBTQ+ themes.

“This is a huge worry,” Wilson says. “We are not America, but we do follow American trends and this is becoming scary.” Intolerance terrifies her. “One of the worst things is there’s no reasoned debate. It’s black or it’s white.” The inevitable result is cancel culture, she says. “People are getting cancelled in cruel and horrible ways. I don’t like the idea of anybody being cancelled because we live in a country where we’ve always taken pride in free speech.” But, she says, censorship is a complex issue. For example, books by Roald Dahl and Enid Blyton have been updated to remove inappropriate and dated language. “Publishers do have a social conscience – life has changed.” She’s happy with this so long as original versions are still available for those who want to read them. Last year, Wilson wrote The Magic Faraway Tree, “inspired” by the Blyton original. She kept true to the tone of the original, but stamped out Blyton’s sexist stereotypes.

Wilson admits one of her own books troubles her today. In 2005, she wrote Love Lessons about 14-year-old Prue falling in love with an art teacher who partly reciprocates. They kiss, and he admits he loves her, too. Would she write the book today? “No. It’s so different now.” She pauses. “Well, I did have my doubts then.” Nowadays, I say, you’d see Prue as a victim even if she had initiated it, and the teacher as a paedophile because he responded to her. “Yes, that’s very true. And it does change things around so much.”

It’s 15 years now since Wilson suffered heart failure, nine since she was on dialysis and had a kidney replacement. “I sat for months in dialysis with Jackie,” Trish says. “It was a terrifying time. Terrifying.” She could not give Wilson her kidney because their blood types weren’t compatible (the only way they do seem incompatible). They joined a scheme for people with a failing kidney and a partner with a healthy one to match other couples in the same situation. Trish gave her kidney to someone in need, while the recipient’s partner donated to Wilson.

As for Wilson’s heart, she’s had a defibrillator fitted. “I’m not going to expose myself, but it looks like I’ve swallowed an iPad.” She giggles. “No, not an iPad, an iPod.

“If my heart stops, hopefully it gives me a powerful kick like a donkey.” She doesn’t write as many words a day as she used to, but she’s still knocking out a minimum of 1,000. First thing in the morning, then in the later afternoon over a glass of wine while Trish is cooking.

We’ve been talking for four hours. Wilson and Trish have been fabulous hosts. We’re standing in front of the epic bookshelf. “I treasure books so much,” Wilson says. “As someone who left school at 16 and never went to university, I am in awe of great literature.” I pull out a Carson McCullers and panic when I realise it’s a special edition.

“Anybody I’ve really loved I get a really special edition. That’s my huge, huge treat,” Wilson says.

What’s the most you’ve spent? “Far too much. Am I going to say that in front of Trish? No.”

“That’s not for public consumption!” Trish says fiercely.

Wilson finds it funny. “There are some booksellers who say my name with such happiness!” she says.

On the way out, I brush past a smiley painting of Wilson. She doesn’t like it, and shows me a copy of one she prefers. “I love this because I look like a serious person. Trish didn’t care for it because I looked quite stern.” She looks at it. “I’m not little Jackie there, am I?” she says. Then she pauses, and laughs. “Oh, but look – he still chose to have the rocking horse in the background.”

The Best Sleepover in the World by Jacqueline Wilson (Penguin Random House Children’s UK, £14.99). To support The Guardian and Observer, order your copy at guardianbookshop.com. Delivery charges may apply.