> There is no equivalent for anything that can be run on a turing machine.
So you're claiming that organisms such as ourselves cannot be run on a Turing machine? Any particular reason/evidence?
> you could write out on paper the calculations that a chatbot is performing to select what it says
In the case of ML, only on the lowest level; there's no human-readable algorithm for putting together utterances. But let's say, for the sake of argument, that someone can actually understand and explain how exactly a particular response was constructed. Could the same not be done, in principle, with the response from a human? You could write out on paper the QFT equations that a brain follows to select what it says.
Well, OK, no one can actually do that, because we don't understand the subject matter sufficiently. But that's my point: That we understand the simpler system better doesn't mean that it's a more abstract concept, or that it can't be mistreated. Of course there is a tendency to have more compassion for more complex organisms — think about how you'd treat a monkey, a rat, a spider, a packet of yeast. We also tend to distinguish between organisms and inanimate things.
But discriminating based on what we understand exactly seems like a bad argument. Also, what if — as seems likely — artificial life will be made like AI currently is? That is: the necessary complexity is too much to plan in detail, so we make a primitive framework in which systems with desirable properties are then bred and evolved. Just like breeding plants and animals still is a thing — we can now sequence entire genomes and research what individual genes do, but it's too complicated to be a viable approach when you want a tastier nectarine.
Can someone ‘write out on paper’ how a Waymo tells a pedestrian apart from a mailbox? If no, does that mean the AI is more prone to being mistreated?
> physical compulsions that it needs to fulfill […] be mistreated by denying these needs
What's the difference between a physical compulsion and a utility function governing a physical system? If I deny a paperclip maximiser the raw materials it needs to make more paperclips, it's definitely not going to be happy …