Actually, there are studies[1] showing the opposite: it's difficult for men to think straight when around attractive women.
[1] for example: http://www.sciencedirect.com/science/article/pii/S0022103109...
[1] for example: http://www.sciencedirect.com/science/article/pii/S0022103109...
The counter argument then is that it is because of society that they are weaker. The same could be argued here: it is because of society's conditioning of men that they can't think straight when around attractive women.