This is a minor nitpick, but I'll bite. I realize nobody explicitly called anyone a mysogynist, but neither did I. There is a difference between saying someone is mysogynist the adjective and mysogyny the noun. jacobian inferred that because the OP doesn't care about diversity then he doesn't care about fairness. That is tantamount to charging someone with sexual discrimination, also known as mysogyny.
[Edit: I upvoted king_jester because, regardless of whether I agree with him on all points, his reply was concise, helpful and he wasn't being a dick]