The most important thing LLMs made me reconsider is the nature of thought itself. What does it mean to think when a computer can do it more convincingly than many of the humans I know?
There has always been a difference between actual understanding and just great presentation.
LLMs are so far ahead in their use of language that given a complete lack of understanding they are more capable than humans to fool the listener. But there's nothing substantially new, some people have mastered that skill, too.
People are regularly doing this in the titles of papers --- ascribing words like "emergence" and "world models" with definitions that don't match their informal usage -- purposely to mislead people into mistakenly applying conventional intuitions and make the paper look like it proves more than it does.
Or, at least, that we should say they don't think, because we're afraid what the alternative says about us.
You presumably use these systems all the time, are you seriously saying that they don't routinely make mistakes that humans don't make?
"LLMs don't think because they fail at X" is very different from "the current crop of LLMs don't think because they fail at X".
I have a hard time believing that someone who was shown a sample of these mistakes (somehow weighted by their frequency) would say that applying the word "understanding" to the system is not more misleading than not.
Someone who was shown a sample of these mistakes would maybe say that the word "understanding" is more misleading than not. I could make the converse argument for someone shown a sample of the successes, though. It seems unfair to judge "understanding" just by what the system can't do, and not by what it can do, while not really applying the same rigor to human babies.
This doesn't scream "not understanding" to me:
> If I'm holding three ice cubes, and I give one to a friend twenty minutes later, how many ice cubes do I have now?"
[... a bunch of correct text elided ...]
> In a practical sense, after twenty minutes at room temperature or in your hand, it's realistic to say that you might have no ice cubes left or just a very small amount of ice/water left from the melted ice cubes.
They don't understand precisely because they don't act like humans, which is the only source of intuition for the word "understand".
> I could make the converse argument for someone shown a sample of the successes, though.
It's the routine mistakes that "reveals" that there is no understanding. By your logic you can point to a single success and declare that as proof of understanding. No ordinary person would agree. A teacher who sees a student regularly make errors in reasoning on an exam doesn't say the student "understands" the material.
> It seems unfair to judge "understanding" just by what the system can't do, and not by what it can do,
I'm not talking about "fair" (whatever that is supposed to mean in this context) I'm talking about an ordinary person's intuition about the word "understand".
Or it was, until LLMs came along. Unless your definition of the word "understand" is a tautological "understanding is what only humans do because humans are the only ones who understand".
> A teacher who sees a student regularly make errors in reasoning on an exam doesn't say the student "understands" the material.
Ah, so you don't mean they're unable of understanding in general, but have not understood specific things they make mistakes about? Maybe we can agree there, although I'm still not sure that all mistakes are the same, in the same way that someone being bad at spelling physics terms doesn't mean they don't understand the physics itself.