Why AGI Is Impossible

"Why AGI is impossible" is our premise. Let's remember that when looking at the argument in the post.

Opening statement: "My point is that intelligence can be perceived in things that are not intelligent." - We already have a circular defeat here. If something is perceived as intelligent, it's either intelligent or our perception is wrong. I think that's what the OP means, but they just state AI is not intelligent instead of focussing on the perception issue.

"AGI might never be achieved with current architecture. Sure, depends on the definition of AGI." So you've moved from 'impossible' to 'might be possible' already. And you've stated it depends on the definition. Feel free to define it fairly, instead of prepping for moving goalposts.

"I envision it as remarkable as described by the AI lab CEOs… basically as a God inside a GPU." Oh. You've gone for one of the harder definitions. But since you've invited goalposts to be moved how about we judge AGI on just general human intelligence levels, not a god's. That's the broader consensus on an AGI definition, and no god has stepped up to have their intelligence tested in a standardised manner before so that's a bad metric.

"LLMs don’t know what they are doing." Do we? Soft and hard determinist models imply we don't have much control over our thoughts and impulses. My brain is a black box to me as an AI's working is to itself. I'd even say an AI has a better understanding of it's inner workings. An AI can tell me its token usage, I can only estimate my calorie expenditure for a thought, and I'd probably get it wrong. I think you have some unstated definition of consciousness, and that you think humans qualify it, and AIs don't. I'm not saying AI is conscious, just that we need to define it, and check it applies to humans objectively before we say humans can do it and AI can't.

"LLMs don’t care about the why of anything." Demonstrably false. Though interesting on your phrasing, unsure an LLM can be accused of not knowing what it is doing in one breath, and then be said to have 'cares' for thing in another. But onto the refutation - an LLM is a collection of 'why' statement. Vectorised information is the collection of answers to why, what, how, when, where statements. You're literally talking about training data as a prerequisite for intelligence. LLMs have tonnes of that. If you're saying an entity must consciously gather training data by itself before being called intelligent that's different. And also wrong. Humans are born with intelligence before they start gathering their training data, we have plenty of baked in intelligence. And LLMs can gather more data too, they can ask why, they can 'care' about why. Agentic LLMs especially do this constantly.

"But that why isn’t the major driver in its output, it is just statically there. Actually it is not even “there”, it was deconstructed into numbers / activation functions" The why is literally the driver of the output, in sum, under constraint of other whys, and random temperature controls. And this is literally the mechanism humans use during sleep to transfer memories from short to long term memory. We take external data, condense it, then condense it further into representations of those experiences. The exact mechanics/processes differ between silicon and neuron, but this is why we perceive intelligence in LLMs, they were modelled after our perceived intelligence.

Galton board example - sure, you're highlighting that its the perception that's wrong. If you don't understand what's going in the middle, you can perceive it wrong. And that'd be the point of my response above. Even when we look in the human mind we still don't see or understand all of it. What we do understand we have replicated in machines. To make a clear statement about intelligence we need more understanding. It's not as easy to say an LLM is closer to a Galton board than a human mind.

And none of these arguments go near affirming that AGI is impossible. If they have any value, they'd just say our current approach is wrong. They don't rule out others. So the title argument is wrong, even if there are merits in the content about current LLM perceptions of intelligence.