It's unfair in the way that unattractiveness is always unfair--no matter what you do, you're judged by a much harsher standard than anyone else, and people desperately want to diminish that reality so they can continue to believe in just world fallacies and retain a sense of control about what happens to them. But in the end there's only two ways to respond: understandably bitter anger at the unfairness of the world, or getting over it. Neither will fix the world, but one will make you happy.