trump winning the presidency twice should prove this is 1000% not true
Luckily, once you're in the voting booth, you don't need to argue with someone to vote: all you need to do is make your marks and hand in the paper.
While my personal experience is affected by being in the SF Bay Area for the past couple of decades, it is my experience that over those decades the overwhelming majority of accusations of racism in ordinary conversation are not invoked because of racism, but are used as a shorthand for "Fuck you, I'm done with this conversation and done with YOU.".