Although not the historical reason, a far simpler one is: you can change your mind and ideas but can't change your race.
In addition, it is that behavior and not that belief that gets the racist removed from the premises. You can choose to be quiet about your ideas where they are not welcome, but you can't choose to not be your race where it is not welcome.