I understand your/TFA's argument but I think falls dangerously wide of the mark. I'm sure that the feedback process you describe is real, but I'm also sure that judicial prejudice is far from the only thing feeding the cycle of poverty and crime that plagues black communities. Even if we were to succeed at eliminating judicial prejudice from our training data, the other factors maintaining the cycle would once again lead the ML algorithm right back to p(criminal|black)>p(criminal).

The way to fight the prejudice is not to pretend that math would work out differently if only we had better data / more sophisticated models / etc, it's is to realize that it's OK to discard the result of bayesian reasoning due to factors out of the calculation's scope. Courts discard evidence all the time for a million sundry reasons. Racism / double standards can be another.