In a way we reverted to this situation after women's sexual liberation. Before the time comes to have children, the most attractive men acquire most of the sex. And this is during the time in which women's attractiveness is at its highest.
It is not until it is time to have children that most women settle down with a man of equal attractiveness (and sometimes not even then since the state will step in to provide resources to single mothers).