This strikes me as eerily similar to the argument that type systems are impossible because of the halting problem. It's sort of true in some sense, but not in an even remotely useful way. So it mostly functions as a way of derailing the conversation away of the more subtle distinctions that do matter (e.g., could we design an easy-to-use type system that rules out this particular type of non-termination/other class of bugs).

There's a large middle-ground between "hard code all your own rules" and "completely unconstrained learning". Learning under constraints is not a new idea.

A classical programming analogy to your argument might be "well the halting problem is undecidable so ignore all this high-level language stuff and just go code up your own turing machine; it's the best you'll ever be able to do".

> because you aren't going to find "fairness" in machine learning.

Why not? The human notion of fairness is fuzzy, which is why researchers have provided various formal notions of fairness in machine learning tasks. Obviously, these formal definitions may or may not correspond to your own gut instinct about what is "fair". And there might be friction between different notions of fairness. None of that should be surprising; otherwise, fairness wouldn't be something that philosophers continue to bleed ink about.

But it is equally obvious that for some notions of fairness, there will exist machine learning algorithms that learn well under the given constraint.