I agree with you that culture matters, and indeed people have been victim of discrimination countless times throughout mankind history. But you are afraid to admit that other aspects might matter too, because it touches your taboos.
You pick the example of black people being thrown to the bottom of the social structure of north american society because you know that arguing with that is unethical by pretty much any current standard. But just recently James Watson stated precisely that the average black person is less intelligent than the average white. Now, quite frankly, I don't care about that subject, but would dare to discuss it or would you just claim that discussion is out of your ethical boundaries? It's not a nice topic to discuss, but because of that you can't just pretend there's nothing to discuss.
I'm just saying: If your arguments are limited by taboos then your opinion can't be of much value.