So it doesn’t matter what the substrate is. The question is not , “is this a turning machine”, or “could you write out on paper what the calculations are being performed”. Indeed if you had perfect insight into all neuronal activity you could probably print output the activity generating thoughts leading to a response.
Instead the question should be, does the entity performing those calculations have sentience. Do those calculations provide the emergent property that allows it to sense and feel, and can it reflect on that sentience.
It seems unlikely that these chatbots do, but that’s not because it’s just a computer. Instead the neural networks don’t seem to be complicated enough to support sentience, and more than likely we are seeing only a very good illusion of sentience.
However, let’s be careful when we asses these things. Although given human treatment of animals, never mind each other, I’m not super hopeful.