See, here is the thing. What you said is completely true, some people grow up in a really shitty environment and the achievements of children needs to be compared with the achievements of those around them.

However, without going into details, I have spent enough time at one of these ivy league universities to see first hand how it works it practice. The black people I met who benefited from these affirmative action policies were some of the richest most privileged human beings in the country. These policies are in no way actually implemented to help disadvantaged people.

A simple test would be your family income, for example. But MIT openly avoids this. These universities definitely don't want "those lower class people" corrupting them with their lower class ideas. They just want to feel virtuous, and the best way to do that is to have people there that "look the part" whether or not they are actually disadvantaged.