The boy who cries wolf is definitely in the wrong, that much is true... But that doesn't remove blame from the townspeople who ignored his cries. It's not a binary absolute, where one person being wrong absolves the other of any wrongdoing.

Intelligent, moral people aren't obligated or forced to jump into action when someone makes an accusation of racism. We can take a moment to consider the available facts, the context, and decide whether the charge of racism seems credible.

It does take effort, so there's a limit to how much attention we can give to these things... But we can also respond to the false accusations, and push back on them for being distractions and wasting the patience of people.

We get the society that is as good as we choose to put in effort to make it.