Yes, and to poke at this a bit further, we could ask "What is it that humans are doing when talking that isn't just simple tricks?"
If I had to hazard a guess, I'd say the answer to that is some sort of persistent world modelling, which means an AI may need to have a sense of self before it can move beyond simple tricks.
However if you construct a ludicrous question it will soldier on mindlessly trying to answer it. For example if you ask something like, if the Golden Gate Bridge were to climb the stairs of the Empire State Building, how many flower petals would it need? The chances are the model will give you a number.
It often says something like that when faced with nonsense, so in some ways it's better than a more 'powerful' program.
https://www.economist.com/by-invitation/2022/06/09/artificia...
>People who interact with gpt-3 usually don’t probe it sceptically. They don’t give it input that stretches concepts beyond their breaking points, so they don’t expose the hollowness behind the scenes.
> “it’s all about the prelude before the conversation. You need to tell it what the AI is and is not capable [of]. It’s not trying to be right, it’s trying to complete what it thinks the AI would do :)”
He did some "prompt engineering" and came up with:
> ‘This is a conversation between a human and a brilliant AI. If a question is “normal” the AI answers it. If the question is “nonsense” the AI says “yo be real”’
which lead to better results. Here is an article about these "Uncertainty Prompts":