That's the voting rights part. Freedom of speech gives you the right to tell your representatives how you want them to represent you. If you choose not to get involved in a contested issue, your representatives will think that the issue is not important to you. They will then prioritize their efforts accordingly. As far as everyone else is concerned, an opinion you don't act upon is indistinguishable from not having an opinion at all.

Freedom of speech makes representative politics a market, where effort is a key currency. You may dislike markets in general or think they are inappropriate in some situations, but they are an inevitable consequence of individual liberties.