But affirmative action doesn't recognize merit, it rewards luck (at being the minority given the job) and is designed to even out ethnic/sex ratios without regard for productivity.

Also, a meritocratic system only requires the "winners" are chosen on the merit of their work. It will never be possible to eliminate all disadvantages which might prevent somebody from achieving some abstract concept of potential.