What test would you propose where an LLM (or an RL agent containing an LLM) would reliably fail, but where an average human would reliably succeed?
I’ll say it again:
> I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
You couldn’t even do it in your counter-example:
> (…) testable characteristic (at least within certain bounds (…)
Even with a much simplified use case, you still had to add the caveat. That should tell you this is much more complex than arguing with strangers on the internet in search of a “gotcha” moment.
Can you, in a short HN moment, propose a good test to (say) prove or disprove the existence of God, depending on your belief? Of course not, because by the very nature of the definitions we have for God, I could always add a counterpoint to make your experiment fail. That would allow me to ridicule and dismiss your position, whatever it is, because it would be framed within unreasonable constraints. I’m not interested in such dishonest conversations, I don’t see winning arguments in itself as a laudable or interesting goal.