Comment by gjstein | Hacker News Reader

Comment by gjstein | Hacker News Reader

> “We need”, Jure said emphatically, “to step up and come up with the means to evaluate – vet – algorithms in unbiased ways. We need to be able to interpret and explain their decisions. We don’t want an optimal algorithm. We want one simple enough that an expert can look at it and say nothing crazy is happening here.

As someone who works with machine learning, I find this statement rather misleading. The researcher quoted here has a very strong bias about what it means for an algorithm to be "optimal". A system cannot both "be optimal" and also "biased" in a way that the designers don't like: such a system is indeed not optimal. The dialog in the machine learning community has increasingly been about how we might structure these systems in a way that they are unbiased; it's a shame the researcher (and the author of this article) seem to think that "simplicity" is the only option.

(In case anyone is interested, I recent wrote a blog post to this effect: http://www.cachestocaches.com/2018/7/bias-and-ai/ )

Every model designer is going to come to the table with his own set of lenses and biases based on this own life experiences. These biases are incorporated into our models based on the metrics that we optimize for and the features that we incorporate into our models.

It may comforting to think that all our biases are “baked out” of the models during the training process but if the researcher had been a black man do you believe that they would have arrived at identical models?

Even if the model designer found ways to compensate for his own bias the world that these models operate in are inherently biased. Training data may be subtly biased in ways that are difficult to detect. A model trained on recidivism is likely to be biased if, for example, a lower class black male or upper class white women have different risks of being arrested for the same crime.

Assuming the models themselves are unbiased as trained they may create bias with subtle errors when pushed into production or due to second order effects, feedback loops or errors or manipulation of data. For example during the credit bubble entire industries found ways to raise borrowes FICO scores.

There is also the inherent corporate bias that the researcher pointed out which the bias of the almighty dollar. Compensating for bias is expensive and may even impact salability if an unbiased model doesn’t “gel” with the expectation of customers who are biased. For example a judge may feel pressure of homogenize his decisions or abdicate his responsibility by rubber stamping the models decisions.

Currently we compensate for this in a democratic society with checks and balances. What are the checks and balances for a computer algorithm that is a trade secret for a for-profit corporate entity?

If the system being modeled is abstract, you might get away with claiming that it is unbiased. If the system being modeled has very much to do with the real world, the modeler is forced to make assumptions about said putative "real" world, because the real world is inconveniently large and complex to fit in memory. However, one of the problems that societies and maintenance programmers encounter is that in the long run these assumptions are invalid.

We've pretty much all read the article "Myths Programmers Believe about Names", and various snowclones. We still write dumb name validation code every day, because there actually isn't a perfect way to do that, and we have to ship something. Even when the concepts are perfectly executed, the fundamental assumptions of the model may change. Having separate classes for "tool" and "weapon" may make sense one day, and the next day you need to write the game Clue. Even if you're working with such boring concepts as filtration, you may have to adjust your model when you discover that water is compressible.

All models are wrong, in many ways, both subtle and overt, both currently and in the future. Some may be useful. I'm not sure that there's any solution to these problems except more and better models though.

Isn't this more often called fairness now since bias is already an overloaded term ?

Reply on news.ycombinator.com