Like thinking LLMs aren’t magic* because you utter “it’s just predicting the next token!” I’d argue, only slightly tongue in cheek, that thinking of LLMs as magical leads to more effective use than the predicting-next-token explanation.

See also Frank Keil’s “illusion of explanatory depth.”

* magic not as “unreal,” but in the classical conception of a living magic world where mental intentions can manifest physical realities