While you make a point that it doesn't always have to do with harassment, your second argument is very weak. There's nothing anyone but yourself can do about "[...] a person's perception of herself and what people might be thinking, not the way others are acting."
If your issue lies with what other people MIGHT be thinking, then the issue is within your own self-confidence.
I agree that stereotypes are destructive, but only when they actually are present. One should be careful not to feel threatened by hypothetical thoughts from hypothetical co-workers before the fact.