Another interesting bit of experiment people did when the GPT-4 class models launched were to test out spatial awareness. For eg, you could describe with words a construction made of blocks, spheres and so on and then ask questions about the stability of the structure. The newer models more often than not got it right.
Of course, that hasn't stopped people from making tired old claims of "stochastic parrot" and so forth but I think if your model can solve such reasoning problem (spatial or otherwise) then there's something really interesting to discover. I'm glad that folks at Anthropic are really trying to create a better understanding of these models (https://www.anthropic.com/research/mapping-mind-language-mod...)