Supporters of slavery used the Bible to justify their beliefs. They did not believe it was about bigotry. They thought slavery was in fact providing economic opportunity to slaves, and they believed their own livelihood depended upon it. Supporters of slavery thought about it in economic and religious terms. If the CEO of my workplace believes in slavery but truly doesn't believe it is immoral, do I not have the right to speak out?
If you believe in slavery or nazism, that's fine. You're allowed to believe in those things. But we are allowed to express that you are wrong for holding those beliefs, and we can choose whether or not we support bigoted viewpoints by purchasing your products. Using one's right to free speech to oppose bigoted speech is not impeding anybody's rights.