The interpretation of the LLM having a "world model" is a big stretch in terminology. Encoding a coherent or accurate set of spatial coordinates for places is not equivalent to understanding the actuality of world space and the relationship between items in that space. By this definition an accurate spreadsheet of the lat-long of every major city would also constitute a "world model". It shouldn't be surprising to anyone that a model trained on (likely many) such spreadsheets would also encode that coordinate data. What's notable is that it still takes a team of human researchers to plot these raw numbers onto a map and to interpret them as experiential differentiators in physical space.

There's no doubt that an LLM can uncover whatever is structurally included in its training data, even if that encoding is implicit. What's less believable is that the LLM somehow achieves a grounded understanding of physical aspects of experience purely from streams of raw text.

I think the paper engages in some equivocation here, as the abstract differentiates between "an enormous collection of superficial statistics" and "a coherent model of the data generating process", without admitting that the first should well imply the second. But that doesn't then further imply a "world model" in the sense that we understand it as sentient beings. For us, an internal model is useful, but when our model or texts don't agree with the actuality of the world, the actuality of the world takes precedence. For an LLM there is no distinguishing between its trained representations and an actual exterior place, or even any sense that an exterior space exists.