As a vegetarian who makes a point of being low key about it, not bringing it up unless asked, etc, there is almost endless judgment and questioning put forth in my direction. Every time I go out for a work lunch and order the black bean burger, or meatless sides at a BBQ joint or whatever, people want to dust off their freshman philosophy class ethical arguments.
People with non-normative beliefs stand out from the normative culture just by virtue of existing. If we must defend ourselves constantly it sounds louder. I understand that can be annoying to people like you.