It's not a dichotomy between "unfair" and "absolutely fair." Things can be more or less fair, and it makes sense to discuss making things "more fair" or "less unfair" even if in doing so, we cannot reach "perfectly fair for everyone, all the time."

Some disabilities cannot be overcome by public acceptance and accommodation, and some can to varying degrees, eg. the Americans With Disabilities Act.