If everyone was perfectly equal in every way except appearance, then it would just be the most attractive men forming harems instead of the most wealthy.
Society is so incredibly complicated and interconnected, it's pretty much impossible to say what would happen by changing something.