Because at issue isn't any ban on free speech, it's whether it's OK to ban invited speakers expected e.g. to incite harassment, bigotry, or violence. And obviously there's a grey area there, but it's certainly no restrain on the message at all - merely the platform (not that that's irrelevant, but the distinction isn't irrelevant either), and I kind of doubt you'd want to provide a platform to everything regardless of what kind of hate speech they're spouting. There's a fine line between a constructive diversity of opinions, including ones you don't agree with, and giving a microphone to those saying nothing new, where the primary draw isn't the message itself, but the drama and tribalism surrounding it.