You are a man and, like all men, need to refrain from imposing yourself on women's spaces. If you, and other men like you, stopped doing that, then there would be no problem. This is an issue that is entirely behaviourally self-inflicted. All you need to do is respect women's boundaries.