Sure, but at that point you're just saying that a neutral descriptive term is more commonly-applicable to one gender.
Is the word "sobbing" or the word "weeping" derogatory? Visibly-emotional crying is also associated with women, and isn't a stereotypically "manly" thing to do.