Being able to kick the idiots out and not compelling other people to amplify their nonsense is a basic mechanism of any society that wants to function.
Also the free exchange of ideas isn't a prerequisite for a just world, it's the consequence of a just system. You can "free exchange of ideas" yourself into a gulag if the population is dumb enough. Exchanging ideas is a process that can go as wrong as it can go right, it's a process, not a set of values.
What built the just institutions you like in the US were semi-aristocratic founding fathers, I went to school in a school system that was built in Prussia, colloquially called "a military with a country", and the civic code in France was written by Napoleon. People in Singapore are prosperous because Lee Kuan Yew had his wits together, not because they enjoy a lot of free exchange of ideas.
It's sane and capable elites building lasting institutions that afford you the luxury of free exchange of ideas, even if this offends the egalitarian myth that is popular in the US. Sorry to say it, but mom, pops and Donald Trump being able to post on Twitter doesn't built functioning states.