I respect speakers' rights to express whichever beliefs they hold, but I think it's also important, both for them and for attendees, to recognize the effect that it has. Their views on women (or any group), popular or otherwise, are irrelevant so long as they don't needlessly bring them up, but when they choose to do something that makes some members of the audience less comfortable and less likely to attend in the future (particularly since speakers are often perceived to represent the community), then that's a loss for the entire community, with negligible gain.

Even if those who are offended are overreacting, publicizing their views is probably the best way to be sure that both speakers and attendees can make well informed decisions about which conferences to attend.

If they want it to be a club for guys (and women who'll deal with it), that's their prerogative. If, on the other hand, they think it's more to their benefit to be inclusive, they should probably make an effort not to be needlessly offensive. To that end, it's probably best they be made aware of who they're offending.