Of course maybe I'm wrong and it's AGI and it will find this comment and torture me for for insulting it's intelligence.
No, please keep it up. Someone needs to keep pushing back against all the "I don't understand it, but it says smart-sounding things, and I don't understand the human brain either, so they're probably the same, it must be sentient!"
It's a pretty handy technology, to be sure. But it's still just a tool.
This perfectly summarize so much of the discourse around GPT.
Except people lack the humility to say they don't understand the brain, so instead they type "It works just like your brain," or "Food for thought: can you prove it isn't just like your brain?"
We don't have to fully understand the brain, or fully understand what LLMs are doing, to be able to say that what LLMs are doing is neither that close to what the brain does, nor anything that we would recognize as consciousness or sentience. There is enough that we do understand about those things—and the ways in which they differ—to be able to say with great confidence that we are not particularly close to AGI with this.
Basically, due to it's nature ChatGPT cannot repeat things verbatim, so it rephrases it. In humans we associate the ability to rephrase stuff with the understanding the material as opposed to rote learning, so we transfer the same concept over to ChatGPT and it suddenly appears "intelligent" despite having zero concepts of whatever stuff it spits out.
I think almost all in HN space would confidently assert that there is no AGI lurking in GPT4+. But add the right higher order modules and self-controlled recursion and Bingo.
It's weird it works when you know how it works.