Google gave this statement to CBS: "Large language models can sometimes respond with non-sensical responses, and this is an example of that. This response violated our policies and we've taken action to prevent similar outputs from occurring."

I think they would have mentioned if it were tricked.

https://www.cbsnews.com/news/google-ai-chatbot-threatening-m...