I'm white yet I don't feel "unfairly tarnished" by the banning of white supremacy.
Nevertheless, plenty of Americans (mostly but not exclusively white) do oppose legislative constraints on white supremacy and I think it's fair to say that they "feel tarnished" by them. My read is that this opposition comes from the same place.