The entire conversation always involves this endless spiel about how GPT is just generating plausible-sounding text and doesn't "really understand" anything. This is not really true, but the larger problem is that it's often left as a purely philosophical point with infinitely high goalposts. What would an AI need to be able to do for people to deem it has this magical property of "really understanding" something? This is often left unexplained, in the sense that it is truly debatable if AI can every "really understand" stuff and... blah blah blah. It's a low-information discussion, in my view.
But GPT really does "understand" stuff in the sense that it does form an internal representation of ideas and concepts in its own latent space, and this is robust to a wide variety of transformations such as variations in wording. It is able to translate between different realizations of the same idea by expressing that idea in different ways: written in English in different styles, written in other languages, rapping; it can even "write a song in object-oriented Python" if you ask it to, and it will do it, and then be able to translate that to an English-language song that rhymes with the same meaning. The fact that it has a many-to-one representation of realizations to vectors in its latent space would seem to be a really good way to quantify what "understanding" means, and that is what GPT is doing.
I think as time goes on, we will get a "human of the gaps" phenomenon with these models.