So when a speaker is invited who is, for example, blatantly hostile to the advertised mission of the university, it raises eyebrows. And it paints a contradictory pictures that devalues the university itself. Or worse, if the message is blatantly hostile to a subset of students, it could discourage some people from even wanting to attend!
People work hard to get accepted to universities which they respect, and they have a vested interest in maintaining their university’s respect in the public eye.