> What would an AI need to be able to do for people to deem it has this magical property of "really understanding" something?
Solve math problems, for one. If you give an AI a simple 3-digit addition problem, it will certainly fail. That's simply because it's model sees "numbers", and instead of processing the context and operation, it will just fill in the blank with more numbers. In that sense, AI is incapable of 'real' thought - the same applies everywhere to varying degrees of effect.
GPT is certainly impressive, but the impressive part is how well it fills in the blank. If it has any ability to reason on it's own, it hasn't demonstrated it reliably yet.