The people also chose the government doing the banning. I don’t find this reasoning holds any water.

If we all vote to ban the party some dude started to advocate for killing all left-handed people, we’re not being anti-democratic - we’re being serious people. If you think this party poses no threat, then that’s a real argument. “But it’s not fair that I can’t vote for the cannibals/Nazis/anti-democracy advocates” is unequivocally not (and it’s not more so just because you don’t think they’re actually Nazis or whatever).