The befuddlement goes even farther for me. LLMs are, effectively, black-box systems that interface with the world via a stochastic parrot interface.

You'd think that software engineers would be a group that easily understands how making radical assumptions about implementation details when looking at nothing but an interface is generally misguided.

I'm not saying that there isn't a strong case to be made against LLMs being intelligent. It's pointing at the stochastic parrot as evidence enough in of itself that confuses me.