Contra Weyl on Technocracy

I loved reading this earlier!

> well - have you heard the story of statistical prediction rules in medicine?

That just crossed my mind recently with all the talk of bias in AI/ML applications (let's not say "algorithms"). Horror stories abound, and I do realize today's systems are vastly more complex & less (to use a term from this post) legible to humans trying to reason about what's going on, but somehow it seems like this aspect is omitted from many discussions today, whereas previously taking the human out of the equation was seen as a success story in some fields.

There's probably also other issues, e.g. that if you're not careful ML can overfit to things that have nothing to do with what you're looking for (e.g. all your positive example photos were taken with a different camera than the negative ones, or what have you), which aren't as big a deal with those simple rules from medicine. But I feel like I'm not really clear on the state of this discussion.

If anyone knows a good paper/article/book that deals with the tradeoff between potentially embedding some form of bias in these systems vs. the also very real biases of, you know, human decision making, but focused on contemporary ML, I'd love to read up on this.