At other times, in other situations, I have been sympathetic to this line of reasoning, but I don't think it applies here.

Let me quote something I wish I had written myself:

When you're offered a chance to get something you want by throwing minorities to the wolves, and you take it because hey, no skin off my nose, that's racism. That's evil. It's low-key evil distinct from actively advocating slavery or painting swastikas on walls, but it's more than mere "apathy" to racial issues.

To actively vote for a racist candidate with a racist platform is simply a racist act, regardless of the voter's reasons or intentions. That is, as far as I can see, merely factual.

"Labeling" is ironically and conveniently a label itself. Sometimes "labeling" is a real concern. This isn't one of those times.