There is a leap from "predicting the next word" to "modelling the world" that you are sort of glossing over here.

The idea that LLM's are actually "reasoning" rather than just performing a probability function seems a little bit...pseudo-religious. Like we've created a new life form.