I think the clear winning strategy is to find some of the common sense reasoning fails that are still quite common, like the "should I walk or drive to the car wash" one from a few months ago.
I think the clear winning strategy is to find some of the common sense reasoning fails that are still quite common, like the "should I walk or drive to the car wash" one from a few months ago.
The reason for the mistake is irrelevant here. All that matters is that you catch the agent saying something that basically no human would, especially in this type of game.
That shouldn't be an issue. The very act of taking the car to car wash contains the information that it is with you and you are at your home. Doesn't this prove that LLMs don't actually think?
Another approach could be to probe the helpfulness of an AI. A human would likely not comply with a request to write a sonnet about tangerines.
And no, I don't think humans would make the same errors, the key reason for this failure mode is that there are some facts of human day to day life that are so mundane that they are virtually never written about, thus making them an ideal combination of universal human experiences but out of distribution for the LLMs.