Predicting the outcome of an experiment. A person/thing who understands something, can predict outcomes to a degree. A person pretending to understand cannot predict with any degree of reliability.
LLMs are literally predicting the outcome of an experiment really well constantly, yet they are best described as pretending to understand really, really well ...
Newton's theory is ultimately and slightly wrong but still super useful and many LLMs are basically like this. I can see why all this becomes confusing but I think part of that is when we anthropomorphic words to describe these things that are just math models.
Science is explanatory, not "predictive" -- this is an antique mistake.
As for 'math models' insofar as these are science, they arent math. They use mathematical notation as a paraphrase for english, and the english words refer to the world.
F=GMM/r^2 is just a summary of "a force occurs in proportion to the product to two masses and inversely in proportion to their square distance"
note: force, mass, distance, etc. <- terms which describe reality and its properties; not mathematics.
Take for example eg continuity. Students are first taught it means you can graph a function in a single stroke of the pen. Later comes epsilon and delta, this is more abstract but at that stage the understanding is that "nearby values map to nearby values" (or some equivalent).
If the student dives in from there, they're taught the "existence" of real numbers (or any mathematical term) is rather a consequence of a system of symbols and relations that increasingly look nothing like "numbers", instead describing more of a process.
Later that "consequence" and "relation" themselves are formalities. "Pure" math occasionally delivers strange consequences in this sense. But it always boils down to a process that something or another must interpret and carry out.
So I wonder whether the edifice is meaningfully a thing in and of itself. Methods developed in ancient China and India etc would have been useful to the Greeks and vice versa, however all of them though worked by means of the human brain. "Line" has a distinct meaning to us, the axioms of geometry don't create the line, they allow us to calculate some properties more efficiently. We always need to interpret the result in terms we understand, don't we?
I see no distinction. If something can explain it can 'predict'.
Humans who just try to sound like an expert also make similar alien mistakes as LLMs do, so I think since we say such humans don't learn to understand we can also say that such models don't learn to understand. You don't become an expert by trying to sound like an expert. These models are trained to sound like experts, so we should expect them to be more like such humans rather than the humans who become experts.
I asked ChatGPT the other day how fast an object would be traveling if "dropped" from the Earth with no orbital velocity, by the time it reached the sun. It brought out the appropriate equations and discussed how to apply them.
(I didn't actually double-check the answer, but the math looked right to me.)
It also seems to have a calculation or "analysis" function now, which gets activated when asking it specific mathematical questions like this. I've imagined it works by using the LLM to set up a formula, which is then evaluated in a classical way.
There are limits on what it can do, just like any human has similar limits. ChatGPT can answer more questions like this correctly than the average person could from off the street. That seems like understanding to me.
And I would say that's because of LLMs ability to predict the outcome of an experiment really well.
As for the "pretending", I think that comes from the fact LLMs are doing something quite different than humans to produce language. But that doesn't make them unintelligent. Just makes them not human.
All you need for that is for the AI to talk like experts, not for the AI to be experts. AI talking like experts without understanding much maps very well to what we see today.