I'm genuinely curious. You would be ok working with actual racists who would be happy to see you dead as long as they are capable? Because its just their opinion? I'm guessing you draw the line somewhere.
If they make threats or disrupt the work, sure, that's over the line. That's true regardless of their motivation though.
We are all human beings not robots. We have emotions. You can understand why a colleague wishing you didn't exist as a person is a net negative to the company right? Even outside work people move away from abusive relationships. Not tolerate it saying that's part of living in the world.