Whether you think it's a big deal or not isn't really the point - after all, do you not believe that, a person who expresses some racist belief, even a true one, about (for instance) a Mexican person feels just as justified in the belief?

You can argue about who is actually justified for feeling what, but crucially, the only impact it ends up having is to further embed the feelings. In other words, it is not a solution to these problems, it is a cause.