It's really funny that you would pick an XKCD that is about how sexism works as part of your argument that it wasn't sexism, merely rudeness.

The whole point is that a man being rude about another man in this particular way -- by superfluously mentioning his attractiveness -- is exceedingly rare, while for women it is all too common. One would be an aberration, the other is pattern, and the pattern is why it's at the top of HN.