As much as I flatly do not believe LLMs can have anything close to consciousness, I definitely agree that the argument of them being just a probabilistic word matching engine can also be applied to our own brains lol.
But that of course is only as it relates to language. Which isn't at all the primary function of the brain and driver of it's experience. It's the other way around because it's all of the other stuff that drives language. Considering the human brain a probabilistic word matching engine fails to capture the underlying forces that direct the language output because a human can say completely different things about the same scenario based on a huge number of non-language inputs or factors. Language is just what humans have come up with to communicate all of the other things.
The same applies to LLMs: we know how they produce language, but the underlying processes that enable it may themselves constitute a form of intelligence.
You can apply anything you want. You can also apply the argument that our owns brains are just a block of cheese too, if you want. (Nobody will stop you.)
When I experience this it’s like the conceptual formulation and language expression part of my brain is quite happy to run itself for a few moments before I - the qualia of being me - has decided to experience that moment.
Utterly bonkers. I love it when it happens. It gets me thinking about how weird language and knowledge and consciousness really are.