It took about five years for the cultural zeitgeist of Germany to swing from extremely liberal to extremely fascist starting in the late 1920s. Sure, false flag and other political shenanigans hastened the process along - but it was the culture, or in your words, "the market", that managed to move rapidly to the right quite alright on its own.
Imagine if the nascent nationalist tendencies in the US and Europe really start ratcheting upward in a serious way over the next decade. Imagine that, at some point, "the market" judges pro-healthcare or pro-LGBT ideas to be harmful and best avoided. What then?
My point is that what you've done with your argument is handed Nazis all the moral authority they need to repress your ideas if their ideas become popular.
The argument for free speech is that we pay the small price of hearing vile ideas as insurance premiums against good ideas being censored if the pendulum ever swings the other way.