Freedom of association means that a clique does not have absolute power over you, right? Competition implies that cliques incompatible with most people’s values will wither.
You can possibly live in the same house. It’s a bit like changing your insurance company or gym membership but with much larger consequences, if I understand the concept correctly (I’ve never read the novels mentioned).