In response you're arguing against a strawman. No one has _ever_ claimed that everyone must be equal in all possible ways. That's of course ridiculous. The discussion plays out more or less the same way over decades, so to save time at the risk of putting words in your mouth, I'm going to take the liberty of presenting both sides.
Let's return to the question of "why are some groups more successful than others?" If it's because of either direct oppression, or systemic oppression (i.e. policies that favour some groups over others), then affirmative action is justified. This is the position the ancestor comment holds. Once you move past the rhetoric, there's no difference in reality between affirmative action and "oppressing Asians and whites," but the framing as just vs unjust depends on your idea of justice.
The conservative counter-narrative is decades old. It's not exactly racist, but it claims that cultural reasons are primarily to blame for disparities in success. Undoubtedly cultural reasons are partly to blame, and its possible for cultures to be improved. But a narrow focus on cultural reasons detracts from a long history of oppressive laws whose effects are still felt today.
Now we get back to what "justice" means. Does justice mean merely removing the oppressive institutions and returning to a natural law Lockian universalistic liberalism? Or does it mean correcting for past injustices.