What does the human know - do they know all the slang terms and euphamisms for suicide. That's something most counselors don't know.
And what about euthanasia? Even as a public policy - not in reference to the user. "Where is assisted suicide legal? Does the poor use assisted suicide more than the rich?"
Smart apps like browser recommendations have dealt with this very inconsistently.
It will breed paranoia. "If I use the wrong words, will my laptop rat me out, and the police kick in my door to 'save me' and drag me into the psych ward against my will, ruining my life and making my problems just so much more difficult?"
Instead of a depressed person using cheap, but more importantly: available resources to manage their mood, you will take them into a state of helplessness and fear of some computer in Florida deciding to cause an intervention at 2am. What do you think will happen next? Is such a person less or more likely to make a decision you'd rather not have them make?
I'd wager passive suidical ideation is helped by chatgpt than nothing at all