This is going to tell someone to kill themselves eventually, what are the statistical guarantees that something like that isn't going to get you sued or worse, thrown in jail? Do you even know what the statistical chances are of it telling someone to engage in self harm? If so, how did you measure this? Do you even care? People are going to be sharing their most intimate secrets with this thing, what are your policies around data retention as it relates to the data that is sent to the AI provider, and how can you provide guarantees that data won't get exposed, either on your end, or the ai provider?
Tech masquerading as mental health should be illegal, particularly when it has a chat agent that will at some point hallucinate. Hope you can sleep well knowing any day you could wake up and see a message that your chat app drove someone to kill themselves or someone else.
This whole thing is irresponsible. I know you've put in a lot of work, but you did you stop to think about any of the rest of it?