Yeah, but it is indirectly the biases of the researchers. A researcher is more likely to notice and correct for training data problems that conflict with their biases.
Come to think of it, isn’t that an interesting venue for GAN-esque methods to detect the relation of patterns falling into these categories of biases? Or is that recursive problem? If not, put me in the paper :-)
That's not strictly true. In a lot of cases you start out oblivious to biases in the data, and then when you evaluate the model you notice problems.
But your point about obliviousness to bias is exactly what I'm speaking towards. One might be oblivious to bias that aligns with your own biases, but notice bias that conflicts with it.
That's not a systematic way of tackling bias. I would rather have invested more in creating good benchmarks and norms.