No. I'm sorry, but spending money is not speech. Restricting the amount of money that one can donate to political campaigns would be perfectly fine, and would not infringe upon anyone's rights at all.
Your right, spending money isn't speech. However, preventing someone from buying an advertisement to display a message and they are denied because its deemed 'political' is definitely limiting someones free speech. It doesn't matter if the message is 'America Rocks', 'Shop at this new store here', or 'John McCain sucks please stop voting for him'; if your gonna limit some but not others because the message is deemed political then you no longer have free speech.
Unlike regular political actions committees, which can donate to candidates and must report who they receive donations from in turn, Super PACs are not required to report their sources of income.
This essentially means that people are free to spend unlimited amounts of their own money talking about their favored political candidate, though they can't donate unlimited amounts to the politician's campaign.
I don't have a strong personal stance on this issue, but I consider the ruling fundamentally fair. If I want to spend an unlimited amount of my own money buying advertising space to raise awareness around global warming, I can do that. That's speech, and large-scale speech takes money. I should be able to do the same thing to support a political candidate.
The presidential election isn't won in the town square, where everyone can hear him gather to speak simultaneously. That might work for mayoral elections of small towns, but it does not work for a nation of 318 million people. And besides, who is going to pay to rent an auditorium for a town hall meeting?
The citizens deserve to hear their elected leaders and candidates speak, and it requires a considerable amount of money to reach such a large audience. (Note that donating air-time or website space or advertising for a candidate is tantamount to donating money.)
What is the serious alternative? Political candidates are limited in their ability to get their message out, based on where they can travel (no money for transportation), and the people they can speak to directly in person? How will democracy function at scale? Presidential candidates are pilgrims, traveling the country by foot? They'll have a hard time reaching a few hundred thousand people at best.
If we did not allow political contributions, and we did allow candidates to spend their own money on their campaigns, then only wealthy billionaires would become successful national politicians. Indeed, many successful politicians are extremely rich, like Mitt Romney (estimated net worth: $250 million), and candidates like Donald Trump rely largely on their wealth. Small-time candidates would have zero chance against them if contribution were not allowed, even while billionaires are spending their personal fortunes.
Now, to be a fair, a purely representational democracy could perhaps function on this basis. The people in a local area elect their local leader, who travels to the capital of a territory to mingle with other local leaders, and from them elect a higher leader, and so on, such that at each level campaigning is all word-of-mouth. But that's an idea that I think will fail in reality at the scale of the USA.
The budgeting problems of campaigns are not my concern. The ability of a few wealthy elites to completely dominate the political conversation simply by having more money, and therefore more speech, is.
You do in fact have the right to speak to a large audience, if you can find one willing to listen.