> What does all this mean? Watson may beat us at Jeopardy, but we are convinced we have something AI will always lack: We are agents in the world, whose decisions, choices, actions are made meaningful by the content of the belief/desire pairings that bring them about. But what if the theory of mind that underwrites our distinctiveness is build on sand, is just another useful illusion foisted upon us by the Darwinian processes that got us here? Then it will turn out that neuroscience is a far greater threat to human distinctiveness than AI will ever be.

I'm not sure how someone could believe that AI would always lack these things without also believing in some form of essentialism, basically believing that major parts of the human experience are "magic" somehow rather than derived from physical processes.

Anyway the author's major concern seems to be that neuroscience has demonstrated that desires and beliefs are not directly real in the same way that, say, your brain is a particular collection of atoms restricted to the physical confines of your skull. I suppose this could be very disturbing if you have the sense that your life is significantly more meaningful if all of your desires and beliefs are as real as the ground you walk on rather than being some illusory projection of the brain.

The problem actually arises from the associations that we have with the word "illusion". As soon as we start to think of our beliefs and desires as "illusions", it's natural to leap to terms like "fake", "false", "worthless", "deceit", and so on. In other words, there are all kinds of negative connotations that we associate with the word "illusion", so when we use that to describe core parts of what we think of as the human experience, it makes us feel bad.

But this is simply a limit of our language: what are the illusions of "beliefs" and "desires" anyway? Desires correspond to our experiences; if we want food, that corresponds to a very real physical state of our physical bodies. Beliefs are a lot shakier because they tend to be be built on complex layers of ideas, but they correspond to physical reality as well, whether or not they're particularly accurate. So these things are very useful and important despite being "illusions". And there's a huge difference between these "illusions" and direct falsehoods. So I think the real limitation here is that we don't have language that's particularly well adapted to discussing and thinking about these things.