While you're at least trying to use the term in a positive light, it's not hard to imagine people using the same terms "AI vegan" and "AI vegetarian" in a disparaging way. The implication is "vegetarians/vegans are preachy/crazy/annoying/arrogant" and using that to describe someone who's anti-AI. E.g. "Oh don't listen to Bob, he's just an AI vegan" or "These annoying AI vegans keep protesting outside and blocking traffic". Now vegans/vegetarians are dragged into something they have nothing to do with, AND you've given detractors an easy way to mock you.