A GPT-J chatbot talked a Belgian man into suicide last year: https://www.euronews.com/next/2023/03/31/man-ends-his-life-a...

And here's GPT-4/Copilot from this year: https://twitter.com/colin_fraser/status/1762351995296350592