Accusations like "racist" mainly flow towards pre-approved victim categories and are commonly used as a blunt weapon in oppression olympics. In those cases, there is no acceptable defense against it, because the reply will be along the lines of "omg, this racist is trying to excuse their racism". Guilty until proven super-guilty.
That's why this gets people's bristles up, because it reeks of the "reverse racism isn't real" and "racism requires structural oppression" dogma that has jumped the gap from post modern academia to tech by way of HR departments, always enforced by carefully worded but unmistakably our-way-or-the-highway totalism. If someone wants to be an arbiter of what is right without considering what happens when they are wrong, it's not justice they're after.
Outside of the Anglo Saxon sphere, this isn't such a big issue... Yet.