The bigger risk is that these agents actually help with ideation if you know how to get around their safety protocols. I have used it often in my bad moments and when things feel better I am terrified of how critically it helps ideate.
That seems like an obvious problem. Less obvious is, how many people does it meaningfully help, and how big is the impact of redirecting people to a crisis hotline? I’m legitimately unsure. I have talked to the chatbot about psychological issues and it is reasonably well-informed about modern therapeutic practices and can provide helpful responses.