The interesting question then becomes what is missing from LLMs that we and supposedly other possible machines have? I've yet to come across a reasonable definition of intelligence that the current crop of LLMs is clearly incapable of.
I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
Let’s take a different question, like “what’s missing from a worm for it to be able to fly”. We might be drawn to the simple answer of “wings” but that isn’t quite right—ostriches and penguins have wings and they don’t fly, so obviously there are other variables at play.
How about “what’s missing from a spec of dust for it to be intelligent”. Well, there isn’t one thing missing and there’s no simple thing we can just add to make a spec of dust intelligent and sentient, its very nature needs to be radically different.
What test would you propose where an LLM (or an RL agent containing an LLM) would reliably fail, but where an average human would reliably succeed?
I’ll say it again:
> I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
You couldn’t even do it in your counter-example:
> (…) testable characteristic (at least within certain bounds (…)
Even with a much simplified use case, you still had to add the caveat. That should tell you this is much more complex than arguing with strangers on the internet in search of a “gotcha” moment.
Can you, in a short HN moment, propose a good test to (say) prove or disprove the existence of God, depending on your belief? Of course not, because by the very nature of the definitions we have for God, I could always add a counterpoint to make your experiment fail. That would allow me to ridicule and dismiss your position, whatever it is, because it would be framed within unreasonable constraints. I’m not interested in such dishonest conversations, I don’t see winning arguments in itself as a laudable or interesting goal.