Innocent until proven guilty. Every citizen has rights until there is reasonable suspicion, probable cause, etc that they are commiting a crime.

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated...

How many of them drank water and/or respired oxygen?

The false positive rate of the test that you suggest is frankly atrocious.

I think I pretty much agree with you that if someone finds evidence of danger to someone else's life then it is a fair expectation that they do something with that evidence. And despite that I still believe that is not a good enough reason to eliminate someone's right to privacy, and yet still that right doesn't mean that we should be able to demand privacy in all products. I think it is fairly clear that these big LLMs are not private products and we shouldn't expect them to be, but I'm still in support of offering private alternatives that people could use without having every input scrutinized.