The University should absolutely be providing speakers that challenge the student's views and offer ideas they may not agree with, but that's an educational decision, not a free speech one.
If the student body wants to lobby the University not to accept you as a speaker, that's their freedom of the speech, and it is up to the University to weigh that against the potential educational value.
For example, to start somewhere obvious, someone coming in to talk about how your friends are subhuman doesn't provide a lot of educational value, and students rightly don't want their tuition spent enabling it.
Don't like it, use your freedom of speech to demonstrate enough value to your ideas the University and student body are willing to listen. Students are at University for an education, and they have a right to say what they find valuable and demand value from the University. Denying them that is limiting their freedom of expression.