Do you think if a non Latin walked into a Latin recording studio that someone would not look at it as not ordinary, or an accordion player walking into a rock studio?
The way to eliminate these tendencies is to make them not exceptional. There was time when Asians were not a majority in science majors... But now they are expected there.
In a perfect world we would not make wrong assumptions, though we'd like the right assumptions.
A fat person might not be presumed to prefer an elevator or a diet soda over a full soda. Or presume American Republicans to be bigots or democrats spendthrifts.
There is a fundamental difference between the experience of being prejudged against when you're a member of a powerful group in society vs. a member of an oppressed group. The difference is difficult for a member of a dominant group to fully understand, partly because they don't have to be aware of their position in a dominant group every single day, whereas members of oppressed groups are constantly reminded of their position.
This isn't to say that men don't have their own issues that they need support with -- issues like (I think) higher rates of unemployment, depression, suicide, drug abuse, imprisonment, lower graduation rates, problems with custody and child support payments, etc. Some people do go too far in saying that if you're a member of a dominant group then you cannot claim to suffer from discrimination in any way, and that's not what I'm trying to say.
I totally agree that the long-term goal should be to make e.g. women in CS programs unexceptional -- and to get there we need to recognize the ways that we make it harder for those women who are in CS programs today to stick it out.