There are people who avoid meat/dairy for their
own benefit. Because they have allergies to it, or have heard eating meat is unhealthy etc.
There are others who avoid them for the animals' benefit, to avoid animal suffering. Because there is no rationale for treating humans' lives as of infinite value, and animals as mere meat machines, fish as just 'stock' for humans etc—except that we are humans and they're not. That naturally extends to avoiding use of other animal 'products' besides food.
Racism and slavery are other phenomena in which an 'out-group' is treated as not worthy of ethical consideration, as not really mattering, just because they're defined as 'not us'. In these situations, there's a similar difference of opinion in just who the life forms whose lives and suffering matter in the situation are. And if you're brought up in a racist or slave-owning tradition, you (at best) resent the hell out of anyone suggesting you that you are unethical, inconsiderate for being that way. How dare they! Who do they think they are, thinking they're better than me! etc.
It will one day seem very odd that nowadays if you abuse and kill a dog or cat you can go to jail; abuse and kill a million chickens or cows and you can be a millionaire.
To return to the point, I've come across both ethical vegetarians and dietary vegetarians (to perhaps coin phrases); some tried going vegan but it didn't work for them; some think avoiding meat is doing enough, it being quite hard to avoid all dairy when eating out. But there are many variations. A lot of people seem to avoid meat, but not fish, even calling themselves vegetarian. I'm kind of 'lazy vegan', I try to never eat/use/buy animal products, without going nuts about it—always trying to ascertain that what I eat from cafes or friends is totally vegan would be a huge pain, for myself and others.