This is a very optimistic world view. If people tend to continuously harrass black people, even for something that is obviously in the nature of the insult, racist (e.g. calling them stupid athletes or something), would you have that same stance? Either everyone is fair game or no one is, so it's okay to keep making fun of that black guy, cause he's allowed to make fun of us as well?
The point is that it's not fair game. This is a conflict involving no women. Yet we still see one being attacked, in a way that you can't really attack a man. You can call a man unattractive, but insults along those lines don't have the same punch, because men aren't defined by the way they look, they're not being objectified. Even when you are using it negatively, it's still objectification.
And the other point is that it's always women. I don't hear very many "your daddy" jokes.