If someone in a conference is open about being a nazi or a racist, you call the police, maybe collect some evidence first, etc. Whatever is needed in your country to report an offense.
And yes, I agree... usually the victim will leave the place, not having enough energy to be an activist on top of dealing with their troubles and/or fearing the consequences. That's unfortunate. But that's where it gets complicated, for the victim. Not for the conference organizers.