It’s an imperfect predictor and Facebook will make a mistake which will eventually be followed by a police shooting. After all, how many stories have been published of police called for mental health issues that resulted in the shooting of the person the caller was worried about.
I hope the programmers are ready to explain how it worked because I get the feeling the family’s attorney is going to have an easy time with black boxes, machine learning, and no true tracing of why.