Prejudice wouldn’t be such an issue if there weren’t a grain of truth in it. Pretty much any superficial indicator that you’ve lived a harder life than average can be used as a way to guess that you might be a worse employee due to some lingering baggage and a less privileged upbringing.
The ugly just sadly don’t have affirmative action to compensate for this sort of thing.