"When you're young, you feel like you should ignore it, because it seems so arbitrary."
Of course different people will ignore this intuitive feeling for different reasons, but I suspect that people rarely do so because it feels arbitrary. Usually people tolerate mean behaviour because there is an asymmetry of power in a relationship. When you need someone more than they need you, they can make harsh demands and treat you like dirt and you will still cooperate. If people refused to cooperate with assholes, or imposed a cost on assholish behaviour then it would stop or be reduced. The fact there are so many giant assholes walking around suggests that being an asshole works, at least sometimes.
I don't think there's anything magical about these instincts. If you observe someone treating another person badly, it's entirely reasonable to imagine that they will do the same to you.