Possible reason being, the more likely you are able to socially connect with your potential victim and the less likely you are to see them as the "other," then the less likely you are to feel justified in causing them harm.
KKK feels justified because other races are inferior, the rich feel justified in exploiting the poor because poor people are lazy, e.g.