Even if we take your claim as true, the fact that different ethnic groups are highly stratified and mobility is restricted, like a caste system, is just as significant. Maybe even more so, because it implies a racist society.
Claiming, in effect, that we should discount a lack of social mobility for those born into certain ethnic groups, seems like racism.