As a man, I've known groups of men to occasionally make crude yet seemingly hilarious remarks that they would not make in the presence of women. If your line of thinking on the subject is correct, I wouldn't be surprised if that was the answer. In the presence of men, other men make men-style remarks. In the presence of women, other women make women-style remarks. I'm not invited to my wife's girls nights with her friends, and she doesn't seem too torn up when she's not on the porch smoking cigars with me and the guys. The reason for that is we want to be able to have gender-specific conversations.
But like you mentioned, if this is happening in the workplace, that's completely wrong for anyone to do. Especially when it's preventing a woman from meeting the former vice president just because she's a woman. Is one woman at a men's only social party a buzz kill? Perhaps. Is one woman at a business meeting a buzz kill? Certainly yes, because there really should be more.