This kind of argument, together with the "Stochastic Parrot" argument falls (in my opinion) under the category of dangerous half knowledge. Somebody read the wikipedia article on LLMs and thinks they know inside out how they work. Whereas the engineers working on these models say that they don't really know why they work / what it is that is manifesting in the different layers during training.

Yes, LLMs predict the next word, but so do we. I, at least, when I reflect on how I form sentences, also start with something and the next most likely word comes out next. You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that.

I'm not claiming that current models have feelings or a deep understanding of the subject matter. What I'm saying is that there's (high likelihood) more going on here then just the simple statistical trick that this article and others like it tend to focus on.