The thing is, the world is full of people who are abusive or terrible when in private or one-on-one, yet adhere to social norms when in a group. Every woman I know who has reported her rapist or her harasser has had this said to her in some variation--it couldn't happen here, because we're not like that. It doesn't work that way.
I've had conversations with male friends about this that eventually boiled down to, "well, I just don't want to think the people I know and like could be capable of that." Yeah, imagine how much we hate it.