There are behaviors that most everyone finds reprehensible, and when someone commits them we want to be able to shun them, to remove our connection.
There are many times in the past when this shunning has been abused to censor or otherwise oppress others.
Generally people feel like they believe the latter part is the most important part...until something occurs in the former part.
As a solution to this dilemma, some advocate strong "non-moral" stances (ala "free speech") where everyone can do as they want, unrestricted. That used to be my answer. But I've come around to believing that if society doesn't apply a means of guidance/restriction/social repercussions the void will be filled with emergent results (i.e. ugliness). Abused rules are abusive, but a lack of a set of rules doesn't lead to peace, it leads to warlords and those least willing to restrain their own actions in the name of empathy gaining the most power.
I don't have the right answers, but I'm confident that walking away from the questions is not the right answer.