Biased Algorithms Are Everywhere, and No One Seems to Care
technologyreview.com
technologyreview.com
There's more to it than the bottom line, the trouble is that nobody wants to take responsibility trying to change the world that they think they've distilled into a model - there's a lot of fitting decisions to past patterns than actually changing society.
The algorithms are inheriting biases from the past and hiding them - your zipcode affecting your repayment potential, that's good old redlining in action all over again. Change can happen, but the algorithms will hide the bias, where it can't be challenged.
And then there's carried-over sample bias - if most people who sign up to clinical trials for cancer are those of a certain level of wealth, with adequate help to drive them to the chemo, then the uncovered genetic samples that come out of the research program will reflect the economic inequality advantages which was offered to a geriatric during their earning years.
Algorithms which rely on past information to be incrementally better are in fact conservative towards their underlying model. Anything else would be self-destructive to the algorithm and in effect, be a problem to the user of the algorithms.