But what does that actually mean? If all you're saying is that the state should not stop them, then relatively few people disagree with that, but the argument usually goes further. There are many ways people's speech can be limited without the involvement of the state. Be that de-platforming, protests or economic or social limitations.
>Restricting Free Speech puts power among the ones in control of Speech.
This is true, but the reality is there will always be restrictions on speech. It is not physically possible to let everyone speak to everyone, or even just those willing to listen. However we structure our societal discourse, it will always privilege some speech over other speech.
We have to engage with, and be ready to criticise, the implicit decisions being made about what speech is privileged and why. Because if we don't, then we cede power to those who already restrict speech with these decisions.
Just saying "don't restrict speech", and thinking that gives everyone a voice, is incredibly politically naive.