What you've said is true, but this is the nature of democracy. A bloc of /any kind/ can vote away everyone else's freedom.
You might not support that, which is reasonable, there's a lot of Libertarians in tech and I do understand where they're coming from. http://i.imgur.com/fV2tHeu.jpg illustrates the point quite nicely.
However: most people don't understand or agree with this, and believe in democracy and its compromises, including shitty things likes rules that may restrict you based on a majority of opponents.