Here’s my anecdata: Tried it with a few different wordings and GPT-3.5 got it wrong every time. 0/3. GPT-4 got it right 2/3 of the times. But if you really want meaningful data I’d suggest reading the paper and specifically paying attention to the human-level performance (95%) of GPT-4 on the HellaSwag data set which includes a lot of these sort of “common sense reasoning” problems. No other LLMs or specialized models have been able to achieve this kind of performance; the state of the art at the time of publication was around 85%.

I'm confused. The claim is that GPT-4 has a world model, not necessarily that GPT-3.5 does not. All that tweet shows is that GPT-3.5 can also give a correct answer. How does this invalidate the author's original claim?