If you ban people because you disagree with them then you don't have a community.
Allowing everyone and everything, all the time, is the exact opposite of a community, it's simply a crowd. But even in a crowd, there are limits to what people will put up with.