Out of curiosity, do you also oppose non-flat taxation, affirmative action, minority-owned business contracting priorities, etc.?
Affirmative action is an attempt to alleviate the impact of prior laws and practices which made people unequal.
I am generally fine with trying to redress previously-enforced inequalities, and suspicious of anyone who isn't (protip: Google "libertarianism, starting right now" for an explanation of why this is important that you're more likely to agree with if a "LOL look how hard I rekt that dude by shoving affirmative action in his face" type sound-bite is the first thing you reach for in this kind of discussion)