However, you're being disingenuous. He didn't wear it at a large event, he thought it would be a fun idea to put it on in front of Trump Tower and share a picture of it with his friends on social media. That's something I might even do as a fun tourist picture, and I really don't like Trump.
This conference is not banning him for conduct at the event, they are banning him for an innocuous social media post on his personal twitter account, a week before the event has even begun. They are just policing the political views of attendees. That's what people have a problem with.