But LLMs do not solve natural language understanding in any of the meanings that the phrase meant before LLMs. Instead, they throw a completely new technique at it that completely sidesteps the need for language understanding and what do you know? For the purposes of responding in meaningful, generally sensible ways, it works amazingly well. And that is incredibly cool. But it doesn't solve the (all) problem(s) that more historical approaches to machine language "understanding" were concerned with.

But there is no world representation inside an LLM, only text (words, letters) representations, so nothing the LLM does can be based on reasoning in a traditional sense.

I would wager that if we build an LLM based on a training data set collection, and then we rebuild it with a heavily edited version of the data set that explicitly excludes certain significant areas of human discourse, the LLM will be severely impaired in its apparent ability to "reason" about anything connected with the excluded areas. That sounds as if it ought to surprise you, since you think they are capable of reasoning beyond the training set. It wouldn't surprise me at all, since I do not believe that it what they are doing.

LLMs contain a model of human speech (really text) behavior that is almost unimaginably more complex than anything we've built before. But by itself that doesn't mean very much with respect to general reasoning ability. The fact that LLMs can convince you otherwise points, to me, to the richness of the training data in suitable responses to almost any prompt,suitable, that is, for the purpose of persuading you that there is some kind of reasoning occuring. But there is not. The fact that neither you nor I can really build a model (hah!) of what the LLM is actually doing doesn't change that.