A friend broke up with her partner. She said she was using ChatGPT as a therapist. She showed me a screenshot, ChatGPT wrote "Oh [name], I can feel how raw the pain is!".
WTF, no you don't bot, you're a hunk of metal!
WTF, no you don't bot, you're a hunk of metal!
An LLM chat bot has no agency, understanding, empathy, accountability, etc. etc.