"If I were I woman (or belong to minority) I would prefer to be being hired without any diversity program."
This is like saying "I'd enjoy my success more without the Mortgage Interest Deduction unfairly advantaging me financially". Theoretically feasible, but in practical reality, uh no. If you're hacking it in a competitive environment and you have a life, you most assuredly have 1,000 more important/interesting things on your mind.