Reverse the races and look at how you'd judge a black equivalent. This one "smallest offense" would color your perception of the person permanently.
"Whiteness" is defined in contrast to a person's blackness. You can't talk about racism against whites without first acknowledging a previous racism by whites.