It's not a lack of "fairness", it's just a lack of accuracy.
Imagine that you train a model to find roofing issues or subsidence or something from aerial imagery. Maybe it performs better on Victorian terraces, because there are lots of those in the UK.
Would you call it unfair because it doesn't do so well on thatched roof properties? No, it's just inaccurate, calling it unfair is a value judgement.
Bias is better because it at least has a statistical basis but fairness is, well.. inaccurate...