If you are "the wrong kind of person," the rules are much less likely to bend for you. If you are the wrong skin color, wrong gender, wrong religion, whatever, going for it can get the rules thrown in your face while everyone accuses you of being an asshole who is trying to break the rules while, at the exact same time, they make exceptions for "the right kind of people."
So, maybe consider that is what a lot of women are complaining about when they decry male privilege.