Please. If you told a customer support rep that you are the former US president [0], they would not hand over the account straight away because you asked nicely.
These models are great tools, but putting them and people on the same level does a disservice to our species and also is simply incorrect to what we know these models to be and their capabilities/limitations.
[0] https://www.theguardian.com/technology/2026/jun/01/meta-ai-h...