The challenge I have when someone calls something racist or sexist, or even good or evil or beautiful or ugly, is that a person seems to be taking how they feel about a thing and labeling that thing as if everyone feels the same way with regards to that thing, which I think is rarely if ever true.
Which can lead to this:
Person 1: That thing is beautiful.
Person 2: No it's not beautiful.
Person 1: Yes, it is beautiful.
Person 2: No, it is not beautiful!
or slightly better, this:
Person 1: For me, that thing is beautiful.
Person 2: Well, for me, that thing is not beautiful.
Person 1: Why don't you find that thing beautiful??
Person 2: I just don't see it as beautiful.
or much better, this:
Person 1: I feel so much joy when I see that thing.
Person 2: I don't feel joy, I actually feel disgust.
Person 1: I'm confused, why do you feel disgust when you see that thing?
Person 2: It reminds me of this one thing that caused me a lot of pain before.
So while yes, I don't think labeling something as universally X necessarily ends the conversation, I think it can lead people to a binary debate about universals instead of open up the conversation for people to share their own experiences and more personal perspectives.