Protesting the speaker's ideas is always fine.

Protesting the speaker's existence is difficult to intellectually justify as a healthy exchange of ideas, including critique.

These days, what we see is a lot of the latter -- up to and including attempting to prevent attendance of the event -- and very little of the former.