Hacker News Reader

Comment by nerdponx | Hacker News Reader

It is precisely systemic racism.

People who do not have dark skin make a system. It turns out that the system performs poorly on people with dark skin, and possibly causes discriminatory outcomes.

Perhaps the training data didn't have enough dark-skinned people. Or perhaps the optics and sensors didn't work well.

The point is that the outcome is biased against a certain group of people, and it's (edit: partly) because that affected group of people lacks representation in the group of people developing the system (edit: , because if they were represented they would have flagged the issue and it would have all been presented and handled differently).

No, it does not imply that the individual researchers are racist as such.

View on HN

Is there a way to prove that it is not caused by systemic racism? Having some unfalsifiable quality is pretty useless, and if it is the sort of thing that's never meant to end then that feels more like religion.

It would depend on the details of the particular situation (was there a specific news story associated with the GP?), but you'd have to show that people with the skin color in question signed off on the product.

It's one thing if you produce a new model and present it at a conference, and one of the conclusion of your paper is that "this model does great on light-skinned faces but performs poorly on dark-skinned faces, we need to make improvements in optics and low-light sensing to bring up the accuracy on dark-skinned people." That's not racist, that's doing science.

It's another if you are a for-profit image recognition company, or if you otherwise deploy that model in production without so much as checking to see if your model generalized well to dark-skinned faces, or without attempting to add some post-hoc bias correction, etc.

We are currently in a world where the latter happens quite often. It's not just a race thing; there all kinds of inadequate "AI" systems being pushed on people that don't really work as advertised, to the detriment of some group or another (sometimes to everyone). But when it becomes a race thing, it's especially distressing to a lot of people, because Europe and the USA have long and grim histories of racism that extend up to and including the present day.

> ... people with the skin color in question signed off on the product.

> ... if you otherwise deploy that model in production without so much as checking to see if your model generalized well to dark-skinned faces, or without attempting to add some post-hoc bias correction, etc

Neither of these would rule out systemic racism in either case, though, since, as you said, the problem is that "the outcome is biased against a certain group of people". Regardless of whether there were BIPOC people signing off or whether you had a perfect training set/etc, the only thing that matters is the outcome.

That's right. My example stated two very different outcomes. One outcome: an academic result is published. Other outcomes: something is put into production.

so you agree that attributing an outcome to systemic racism is unfalsifiable?

I see what you're saying.

Mind you, I'm not a sociologist, so anything I say here is going to be a suboptimal paraphrasing.

Systemic racism is not a cause of something, but a description of a state of existence.

Systemic racism is itself the existence of inequitable outcomes in the context of a society that was literally founded on overt racism.

That something "is" systemic racism is falsifiable by demonstrating that one or both of those criteria is not met.

So this particular outcome will always be definitionally systemic racism until the outcome is no longer inequitable, or the USA has moved meaningfully past its white supremacist origins to the point where individual cases of accidental bias are truly individual cases and not part of a pattern.

Personally, I am okay with that. Ceteris non paribus sunt.

Agreed, but I've found that when discussing these things logic isn't often taken into account anymore due to matters of sensitivity and bias. It's frustrating.

Alternative take:

Computers can most easily pick out facial features when there is significant color contrast, shadows cast on white skin create greater contrast than it can with darker skin, and the systems end up inherently working better on people with lighter skin due to basic math and physics.

Why hasn't some company out there designed a powerful system that works well with darker skinned people? It would be trumpeted by the tech press all over if they did, instead we just see articles about companies like Google manually tweaking the outcomes to appease the people complaining about racism.

A system that works poorly on dark-skinned people would be a nonstarter in most parts of Africa and South Asia.

But in North America it's hailed as highly successful and put into production, despite some 44 million US citizens being Black or African, with demographic concentrations as high as 70-80% in some cities.

From looking at articles that actually give hard numbers, misidentification for minorities is something like 5 to 10 times more common, with a 1 in 1000 error rate for black women for example:

https://www.wired.com/story/best-algorithms-struggle-recogni...

Being able to identify 999 out of 1,000 black women accurately is highly successful. Just because it gets 9,999 out of 10,000 on white women doesn't make 999 out of 1,000 a failure.

Mind you I'm opposed to the tech in general because it will inevitably be used to create dystopian hellholes. But not because I falsely correlate different error margins for different races with "structural racism."

I understand what you're saying, but it sounds like you're aware of how prone this term is to misinterpretation. Virtually everyone outside of a particular small subculture understands "systemic racism" to be an accusation of racism. It would be like calling a correctness bug "systemic fraud" - how are the authors of the system supposed to avoid taking that as a personal attack?

> It would be like calling a correctness bug "systemic fraud" - how are the authors of the system supposed to avoid taking that as a personal attack?

Perhaps they aren't.

Anecdata:

At work one time I created a machine learning "algorithm" that turned out to be systemically biased in a way that was both unintended and distasteful.

I'm sure I've done it other times, but this is the one time that we actually realized it.

The issue was flagged internally and the model was improved until we felt like we were no longer producing bigoted results.

I certainly did not take offense at the suggestion that my model might be racist. On the contrary, I considered it a very important and serious problem that needed to be resolved.

It's worth mentioning that part of the reason we even caught the problem is that some of my colleagues at the time would have been affected by the adverse outcome, had they been subject to the judgement of the model. If I didn't personally work with people who fell into the affected group, I might never have noticed the problem in the first place.

Reply on news.ycombinator.com