I would guess that the key factor is whether or not a body part is significantly erogenous. AFAIK, many more women than men find their nipples to be a source of sexual arousal.
EDIT: I hope people reading my post understand the difference between explaining vs. endorsing another party's reasoning.