What should happen: "I understand that this person has black skin and therefore according to elementary statistics is more likely to be a criminal, but acting on the information would create an unfair double standard that punishes people for crimes they didn't commit, it would help perpetuate a cycle, and it would violate individual human dignity, so I won't."

What actually happens: exactly what parent said. "Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!"

facepalm.

Though blaming algorithms for being "biased" is rather strange as well. It seems to me the algorithms do exactly what you'd expect them to: have predictive capabilities about reality.

Of course, there is a problem here. You don't train neural networks with data of people who've committed crimes. You don't have such data. So what data has been used by the engineers to feed to this machine? Data of people who have been convicted of committing a crime, mostly likely.

And now you've constructed a feedback loop. Those pesky blacks are more likely to commit crimes? Always knew it. A man has been murdered and there are 2 suspects, one white and one black? Why even bother investigating the white guy, we already knew it was the black one. Holy cow, 1 year with the new system and black crime has increased by 50%! Always knew it. Better keep the system in place to save everyone from those pesky blacks.

Oh, the irony.