You're acting as though the likely outcome of neo-nazi groups is some skinhead trying to fight you. What actually happens is neo-nazi groups radicalize angry young men who then go out a gun down a whole bunch of people at a synagogue.
>Trans people are for some reason the only large visible group yelling that speech is violence and saying mean things about them should be illegal.
This is simply nonsense. Pretty much every group in the country has a contingent that wants speech against them regulated