I highly doubt that the preference for male candidates was the only problem with the AI. The preference for candidates with ice hockey on their resume almost certainly also would have resulted in a preference for white candidates.
The correlation with ice hockey could be a career relevant detail because of correlation with discipline and pain tolerance. If it tracks all such signals, I don’t see the problem.
That it also has a circumstantial correlation with races isn’t inherently problematic — it could be that different groups of people are differently qualified.
There seems to be a position that culture is uncorrelated to job success, and that’s just nonsense. When people can show me these biases are correlated to race or sex once controlling for culture, then we have a problem.
That’s why it’s only meaningful to draw these conclusions once you control for cultural variation.
No one has shown that they drew conclusions based on race: they drew conclusions based on written words, which correlated with race. The claim is that this demonstrates racial discrimination.
However, an equally valid explanation is culture as a confounding variable correlated with both race and hiring, and which is the actual causative factor.
My point is that no one has shown this is hiring based on race, because the only things being judged are cultural artifacts. It’s merely presumed that anything correlated with race is inherently racist, when that’s not true.
No one has even remotely tried to address that, including yourself, preferring to merely assume it’s racism or special pleading like “well, there is some correlation — but without doing the math, I know that can’t be it!”
(All these same point apply to sexism.)
We've been using "ice hockey" as an example, right? Given that it's not obviously correlated with fitness for a job at Amazon, it's incumbent on you to show that it actually is. Until then, we should assume that it's at best a false positive from unrelated data (like the ruler in the skin cancer example) and at worst a proxy for race and gender.