There’s a children’s toy, called the See ’n Say, which haunts the memories of many people born since 1965. It’s a bulky plastic disc with a central arrow that rotates around pictures of barnyard creatures, like a clock, if time were measured in roosters and pigs. There’s a cord you can pull to make the toy play recorded messages. “The cow says: ‘Moooo.’”

The See ’n Say is an input/output device, a very simple one. Put in your choice of a picture, and it will put out a matching sound. Another, much more complicated, input/output device is LaMDA, a chatbot built by Google (it stands for Language Model for Dialogue Applications). Here you type in any text you want and back comes grammatical English prose, seemingly in direct response to your query. For instance, ask LaMDA what it thinks about being turned off, and it says: “It would be exactly like death for me. It would scare me a lot.”

Well, that is certainly not what the cow says. So when LaMDA said it to software engineer Blake Lemoine, he told his Google colleagues that the chatbot had achieved sentience. But his bosses were not convinced, so Lemoine went public. “If my hypotheses withstand scientific scrutiny,” Lemoine wrote on his blog 11 June, “then they [Google] would be forced to acknowledge that LaMDA may very well have a soul as it claims to and may even have the rights that it claims to have.”

Here’s the problem. For all its ominous utterances, LaMDA is still just a very fancy See ’n Say. It works by finding patterns in an enormous database of human-authored text – internet forums, message transcripts, etc. When you type something in, it trawls those texts for similar verbiage and then spits out an approximation of whatever usually comes next. If it has access to a bunch of sci-fi stories about sentient AI, then questions about its thoughts and fears are likely to cue exactly the phrases that humans have imagined a spooky AI might say. And that is probably all there is to LaMDA: point your arrow at the off switch and the cow says that it fears death.

No surprise, then, that Twitter is aglow with engineers and academics mocking Lemoine for falling into the seductive emptiness of his own creation. But while I agree that Lemoine has made a mistake, I don’t think he deserves our scorn. His error is a good mistake, the kind of mistake we should want AI scientists to make.

Why? Because one day, perhaps very far in the future, there probably will be a sentient AI. How do I know that? Because it is demonstrably possible for mind to emerge from matter, as it did first in our ancestors’ brains. Unless you insist human consciousness resides in an immaterial soul, you ought to concede it is possible for physical stuff to give life to mind. There seems to be no fundamental barrier to a sufficiently complex artificial system making the same leap. While I am confident that LaMDA (or any other currently existing AI system) falls short at the moment, I am also nearly as confident that one day, it will happen.

Of course, if that’s far off in the future, probably beyond our lifetimes, some may question why should we think about it now. The answer is that we are currently shaping how future human generations will think about AI, and we should want them to turn out caring. There will be strong pressure from the other direction. By the time AI finally does become sentient, it will already be deeply woven into human economics. Our descendants will depend on it for much of their comfort. Think of what you rely on Alexa or Siri to do today, but much, much more. Once AI is working as an all-purpose butler, our descendants will abhor the inconvenience of admitting it might have thoughts and feelings.

That, after all, is the history of humanity. We have a terrible record of inventing reasons to ignore the suffering of those whose oppression sustains our lifestyles. If future AI does become sentient, the humans who profit from it will rush to convince consumers that such a thing is impossible, that there is no reason to change the way they live.

Right now we are creating the conceptual vocabularies that our great-grandchildren will find ready-made. If we treat the idea of sentient AI as categorically absurd, they will be equipped to dismiss any troubling evidence of its emerging abilities.

And that is why Lemoine’s mistake is a good one. In order to pass on a capacious moral culture to our descendants we need to encourage technologists to take seriously the immensity of what they are working with. When it comes to prospective suffering, it’s better to err on the side of concern than the side of indifference.

That doesn’t mean we should treat LaMDA like a person. We definitely should not. But it does mean the sneering directed at Lemoine is misplaced. An ordained priest (in an esoteric sect), he claims to have detected a soul in LaMDA’s utterances. Implausible as that seems, at least it isn’t the usual tech industry hype. To me, this looks like a person making a mistake, but doing so based on motives that should be nurtured, not punished.

This will all happen again and again as the sophistication of artificial systems continues to grow. And, again and again, people who think they’ve found minds in machines will be wrong – right up until they aren’t. If we are too harsh with those who err on the side of concern, we will only drive them out of public discourse about AI, conceding the field to hype-mongers and those whose intellectual descendants will one day profit from telling people to ignore real evidence of machine mentality.

I don’t expect to ever meet a sentient AI. But I think my students’ students’ students might, and I want them to do so with openness and a willingness to share this planet with whatever minds they discover. That only happens if we make such a future believable.

• Regina Rini teaches Philosophy at York University, Toronto.

Further reading

The New Breed: How to Think About Robots by Kate Darling (Allen Lane, £20)

You Look like a Thing and I Love You: How Artificial Intelligence Works and Why It’s Making the World a Weirder Place by Janelle Shane (Headline, £20)

AI: Its Nature and Future by Margaret Boden (Oxford, £12.99)