Sure but they're not all in the same ethnic group. What if a populous group had the peak of their intellectual curve far down the range? If, say, Gauls/French people were almost entirely simpletons equivalent to someone with Down Syndrome? Would they be enslaved and turned into a worldwide servant class? Every other group that were thought of as being inferior can be demonstrably equal to their oppressors but I think a world in which they weren't could be very different.