The problem is, of course, that society still is racist.
Multiple states currently have lawsuits in process against them alleging that they created voting district maps specifically to devalue the vote of black people. Another state just had the Supreme Court rule against them for the same thing.
I don't think the system of affirmative action was well designed or worked particularly well, but it's not true that in the US just saying 'everything is equal now' is going to be enough.