Once you educate yourself with fact-based historical information to understand how unfair the system has been, this actually is an argument in favor of affirmative action.
Point being that until very recently the biases has been extreme, so that despite positive changes in both culture and law, equality will not happen over night (as you said, things persist), and thus affirmative action is important even after the most pernicious source of discrimination has been removed.
Affirmative action is complex, no question, but your particular point is more of an argument for it than one against it.