I'd argue we should strive for a society where the suffering you're exposed to if you're unlucky enough to be in the bottom quartile is bounded, while still allowing for the top to be unbounded in pushing humanity forward. Accepting there's natural variance here is part of that.
We're not all the same, things aren't fair. We shouldn't ignore that or pretend otherwise, but we also shouldn't think that means those dealt a bad genetic hand need to be totally screwed in our society (imo) and it doesn't mean you need to handicap the outliers on the other side in some Bergeron like pursuit of 'fairness' [0].
[0]: http://www.tnellen.com/cybereng/harrison.html
Also related on the prettiness bit, this short story is good: https://waldyrious.neocities.org/ted_chiang/liking-what-you-...