If someone in a conference is open about being a nazi or a racist, you call the police, maybe collect some evidence first, etc. Whatever is needed in your country to report an offense.
And yes, I agree... usually the victim will leave the place, not having enough energy to be an activist on top of dealing with their troubles and/or fearing the consequences. That's unfortunate. But that's where it gets complicated, for the victim. Not for the conference organizers.
But you are making the decision that this is something for the victim to deal with, not the organizers. That's not at all a given by default. As you yourself point out, the same standard is not applied to nazis, so why pedophile rapists? If you are in a position of power it is your job to make these decisions one way or the other, it's not a foregone conclusion.
The relevant quote from The Open Society and Its Enemies is:
> But we should claim the right to suppress them if necessary even by force; for it may easily turn out that they are not prepared to meet us on the level of rational argument, but begin by denouncing all argument; they may forbid their followers to listen to rational argument, because it is deceptive, and teach them to answer arguments by the use of their fists or pistols.
It's about the right to suppress those who leave the arena of debate and move to extra-judicial force, such as the extra-legal violence used to keep black people subdued during segregation (the work was published in 1945, segregation is very relevant).
What you described is people not wanting to work with people that hate them.