If someone is intoxicated, I agree that it seems weird to disregard their consent when drugs+alcohols are the social lubricant of society (and very interwined with sex).
Also, of course you are responsible for your actions even when under the influence (drive and kill someone? no excuse because you were intoxicated - it's your fault). It's crazy to me that people call that "victim blaming". Although I understand how someone can take advantage of others, I don't think the distinction is intoxicated = taken advantage of.