Look at how superior Americans are to Europeans economically. Americans invented the iPhone, Google, and could best all of Europe combined in a military battle. Does that mean that Americans are genetically superior to Europeans? No, it's that they had access to more resources on a bare continent, which led to a positive feedback loop of wealth and creation.
You can perhaps make fuzzy statements about averages or extremes, but what matters is how you act on those beliefs. Are black people better at basketball than whites or Asians? Hard to say on average, but maybe at the extremes? Does that say anything about which races should play in the NBA? No. It's not like Larry Bird or Jeremy Lin don't exist. There might be some differences there, but they're not substantive.
The minute you start using this to justify slavery, that's when it becomes racism. If you are white, would you accept an Asian person's claim to enslave you based on the fact that their IQs are higher on average?
Even if you accept that intelligence implies a right to rule, there are plenty of dumb Asians that don't deserve to rule over a smart white person, and likewise for whites and blacks. This is a simple consequence of the fact that individual variation is greater than group variation.