It seems like those calling for removing speakers are often trying to set identity contingencies. That is, a rule like, "if you are Douglas Crockford, then you can't speak at a conference". However, like identity contingencies against people based on gender, race, etc.. this kind of rule is disappointing because it assumes there is something inherently limited or bad about being Douglas Crockford.

What seems better to me would be asking for an explicit behavioral contingency on being a conference speaker. For example, "if you are going to make sexist remarks, then you cannot speak at the conference".

This makes clear that if you think Douglas Crockford is not fundamentally broken as a human, but could learn and become better at following the contingencies set by the conference, then he is fine to speak. It also opens room to work with Crockford to create a plan for him to speak at the conference, such as going over his speech beforehand, or sitting down with him afterward to discuss ways in which he may have violated the conference's standards.

He can't stop being Douglas Crockford, but I don't think that means he can't stop producing behaviors conference organizers find offensive or harmful (if he wants to).

P.s. check out the book Whistling Vivaldi for an interesting introduction to some potentially harmful (or beneficial) effects of identity contingencies.