I mean, we can get a more real example easily. Amazon had an AI recruiting tool that was biased against women. The explanation is very simple, the training dataset of successful hires mostly consisted of males so the AI though nothing better than penalizing women.
The question is, how do you detect such a bias, and how do you prevent it in the first place? We have some ideas when it comes to humans, but for code it's much more challenging.
https://www.reuters.com/article/us-amazon-com-jobs-automatio...
The subject of AI and ethics is, in my opinion, something that is not considered enough by the actors of the field, including developers.
https://www.propublica.org/article/machine-bias-risk-assessm...
https://www.boredpanda.com/google-translate-sexist/
https://openai.com/blog/multimodal-neurons/#biasandovergener...
https://www.theverge.com/2018/1/12/16882408/google-racist-go...