It is a tricky and difficult subject, I would not be quick to dismiss either argument.
Even worse, it also might encourage more division and more prejudice towards those minority groups. If a company gives preferential treatment to hiring women, then what are many coworkers going to inevitably think when a woman makes a mistake or something? "You only got hired here because you're a woman." And in that case, they actually have some empirical backing for that assumption. I don't belong to a minority group, but I feel like the anxiety of other people wondering or believing I only (or primarily) got into a certain place because of my gender or race, and not my qualifications, would be very uncomfortable and demotivating. Some people may already think that even when it's plainly not true, but a lot more people will think it if it starts becoming more true more often. Not to mention, on top of that, fear that maybe I'm not that great and my gender/race really was a major factor in my hiring decision. Talk about imposter syndrome.
Also, just on a more general ethical note: People in certain groups have indeed been treated very unfairly for centuries, but flipping the table feels kind of "eye for an eye".
Of course unconscious biases are a major problem and will probably be impossible to get rid of or ignore during a hiring process, but I think blind hiring procedures (no name, gender, age, etc. listed on resume or during the first stages of initial online interviews) are an all around better option. I think if organizations work hard enough, they can eliminate a lot of these issues in the pipeline. If, hypothetically, an equal number of male and female candidates make it through the blind stages, but many more men make it through the following stages, they can look at it and quickly determine if they have a problem that needs addressing.
It's especially tricky because the pro-affirmative-action perspective relies on affirmative action actually working. To the extent that it works, the original case for the special treatment weakens. To the extent that it doesn't work, then affirmative action is some sort of token sacrifice, not a way to actually help people. There's an inherent tension in there that most affirmative action proponents aren't really comfortable with.
It also shows up in interviews surprisingly.
He did the same thing to female sales reps - insisted we give them extra help above and beyond what he'd want us to do for men. Very glad I got out of retail hell.