This is precisely why I voted against abolishing cash bail in California last year. Because they were going to replace it with some "algorithm." I hate cash bail, I think it's garbage, but this black box approach is worse.
Algorithms can be okay, as long as they are transparent. The problem with using things like AI is that the parameters or the algorithm are often hidden from the harmed individuals. And the law enforcement community can kick the can down the road by saying "but the software says...".