One example in the article is that if you have 35 sentences leading up to a 36th sentence conclusion, ChatGPT is very likely to shorten it to things in the earlier sentences and never actually summarize the important point.
Just the opposite: it calls into question if _we_ have thinking and understanding capabilities or if we are complicated stochastic parrots. [3] The best probing of these questions is done at the limits of comprehension and with unique and previously unseen information. I.e., how do you comprehend and process to previously unseen/unfelt/not-understood qualia? Not about how you deal with the mundanity of reactions between people (which are somewhat trivial to describe and model). [4]
[1] https://en.wikipedia.org/wiki/ELIZA [2] https://en.wikipedia.org/wiki/Anthropomorphism [3] https://www.newyorker.com/humor/sketchbook/is-my-toddler-a-s... [4] https://en.wikipedia.org/wiki/Games_People_Play_(book)