It is generally used that way. But it's also sometimes used in the gender-specific manner, and so some people will take offense over it.
Well, most of my friends and acquaintances are women and it's not uncommon for them to say stuff like "I need to grow a pair" or "I need to just man up" if they think they're being a bit unconfident. They're very gendered phrases and I must admit to finding it a bit strange hearing women say them, but it does demonstrate that they've become pretty neutralised in modern parlance.
A social justice warrior would suggest that those women are experiencing internalized sexism.