To be fair, I don't think most birds have a good sense of smell (as dogs, cats, deer etc have) so that would discount odour. I suspect if there is a factor here it may be something that is obvious to the birds.
Another hypothesis given is sunscreen. If women wear sunscreen more often (such as on their face) and sunscreen reflects UV (and birds can see in the UV spectrum), then that could also explain it.