There's an important distinction to be made here between public and private colleges. Can't private colleges, being private entities, restrict speech however they see fit, in the same way that your company can?
You might argue that the acceptance of strings-attached public money bears on whether a private college is truly private, but this does not change the fact that a truly private college and a public one operate under different legal regimes regarding free speech.
If colleges can be regulated in that way, then why not regulate corporations the same way? A company like Lockheed Martin probably gets a bigger percentage of its revenue from the U.S. government than any college.