If we somehow discover LLMs right after Newton discovered the theory of gravity, and then a while later Einstein discovers General Relativity, then GR would not be in the training set of the neural net. That doesn't make GR any less of a description of reality! You also can't convert General Relativity into whalesong!
But you CAN explain General Relativity in English, or in Chinese to a person in china. So the fact that we can create a mapping from the concept of General Relativity in the neural network of the brain of a human in the USA using english, to someone in china using chinese, to a ML model, is what makes it a shared statistical model of reality.
You also can't convert General Relativity to the language of "infant babble", does it make general relativity any less real?