If you live in isolation or aren't a member of a group that is perceived negatively by a significant number of people, it's very easy to NOT care. It's a lot harder when how people feel about you affects you - every day - in your ability to get/maintain a job, get promoted at that job, buy/rent a dwelling, finance a business, enter a restaurant/elevator/bus without causing a ripple. . .
Here's a simple answer: "people" are materially dependent on interactions with "other people", and outcome of these interactions is to a large degree determined by how "other people feel about them".