While AI is not a real human, brain, consciousness, soul ... it has evolved enough to "feel" like it is if you talk to it in certain ways.
I'm not sure how the law is supposed to handle something like this really. If a person is deliberately telling someone things in order to get them to hurt themselves, they're guilty of a crime (I would expect maybe third-degree murder/involuntary manslaughter possibly, depending on the evidence and intent, again, not a lawyer these are just guesses).
But when a system is given specific inputs and isn't trained not to give specific outputs, it's kind of hard to capture every case like this, no matter how many safe-guards and RI training is done, and even harder to punish someone specific for it.
Is it neglect? Or is there malicious intent involved? Google may be on trial for this (unless thrown out or settled), but every provider could potentially be targeted here if there is precedent set.
But if that happens, how are providers supposed to respond? The open models are "out there", a snapshot in time - there's no taking them back (they could be taken offline, but that's like condemning a TV show or a book - still going to be circulated somehow). Non-open models can try to help curb this sort of problem actively in new releases, but nothing is going to be perfect.
I hope something constructive comes from this rather than a simple finger pointing.
Maybe we can get away from natural language processing and go back to more structured inputs. Limit what can be said and how. I dunno, just writing what comes to mind at this point.
Have a good day everyone!