Yeah, as someone with an education background I suspect GPT-4 is relatively close to the general public's performance on this problem. Many people would miss AIW, and almost all would miss AIW+. I'm about as good at this kind of thing as anyone and I'd need a minute with pencil and paper to handle AIW+; it's on par with the most difficult problems found on tests like the GRE.

I wonder if these models, trained on data from across the internet, are in some ethereal way capturing the cognitive approaches of the average person (and not picking the best approaches). If the average person does not think in these sorts of symbolic-manipulative terms, and therefore does not write in those terms, and you train a model on that writing...?