> LLM systems are strongly predictive

No, they're not predictive at all. They're generative. A prediction is a forecast based on a theoretical model about a state of affairs in the world. Making a guess what the weather tomorrow will be is a prediction, and a prediction can be true or false. An LLM generates text based on certain parameters, and output it generates aren't true or false, they're simply whatever the system is tuned to produce.

LLMs are terrible at making forecasts or world modelling precisely because they don't actually operate on much of any model or theory, they have no notion of truth or falsehood, they simply generate whatever you want them to generate.