> Thanks for the further background information. I have to say it doesn't really make it better for me. The "angry people" are of course correct that you can also create bias in other ways than data sets. But are they implying that people generally deliberately introduce such biases to uphold discrimination? That seems like a very serious and offensive claim to make, and not very helpful either.

No.

I think Isbell's Neurips Keynote (https://nips.cc/virtual/2020/public/invited_16166.html), titled "You Can’t Escape Hyperparameters and Latent Variables" does a good job of explaining this.

The humans who ultimately validate the model (and who decide on the dataset) are a hyperparameter. Often ignored, yes, but they are still part of the training loop. They decide what the other hyperparams are, when to stop training and publish, etc.

To use a question I've asked on HN before: say you're training a model to detect criminality based on facial structure. This has come up as a real world example, papers have been published on this topic. What does a "good" dataset look like? Or similarly, for a system that decides on bail or sentence length. Do you use historical data on bail or sentencing? We have very well documented examples of bias in both of those things, even in the ground truth. So how do you decide to mitigate that bias? Or do you choose not to, and to continue enforcing said biases in your model?

> But overall, such misclassifications are simply "bugs" that should get a ticket and be fixed, not trigger huge debates

But when such "bugs" aren't prioritized because people don't think they are bugs, you have to debate whether or not they are bugs at all! The hyperparameter here is "who decides what is or isn't a bug"