Everyone decides for themselves.
> who is the ultimate arbiter of acceptable speech?
Ultimate arbiter? Nobody. Please tell me, who do you think is the ultimate arbiter? Do you think one individual in the crowd was the ultimate arbiter? What is their name?
> you might agree with them today but you should be incredibly wary of ceding the power to regulate speech to them lest you disagree with them tomorrow
If a crowd of people were that crazy about getting me to not speak at their college, I would happily leave and find another place, and write a Medium post about my views, not crying that people don't like me and don't want to hear them.
> why is the preference of the protestors more important than the civil rights of the speaker and those who invited them?
Why are the preferences of the speaker and those who invited them more important than those of the protestor? Do you think whoever is called a "speaker" should become "the ultimate arbiter of acceptable speech"? Because I don't. I don't like putting power in one person's hands.