Fighting for the other persons right to speak their thought is a good and noble thing as long as the other person doesn't fight against your right to say what you like and as long as their words don't turn into actions. The more of this happens the more tolerance becomes a luxury only those not directly involved can afford.
Let us substitute something else for right wing/fascist extremism for a moment: Dschihadism. When do we stop tolerating Dschihadists? And when do we start treating every Dschihadist as a potential terrorist, which should be surveilled, silenced, imprisoned and ultimately even killed? The answer is very clear: when the frequency with which people violate other peoples fundamental rights in the Name of Dschihadism starts to rise too much. Be it their children or women who are at times forbidden basic human rights, be it the victims of their violent attacks. This is way beyond being just speech, this is anti-humanistic ideology and it should be stopped just at the point where words become actions.
Now if you ask how things changed so quickly: A significant part of the US public came closer to these actions (with a strongly visible part trying to stop the counting of a democratic election with violence). During the Trump years it wasn't just speech that was problematic. People got killed, people got attacked, people got their rights violated in many ways based on that speech. And there is a threshold beyond which tolerance makes you complicit.
Don't become the kind of person that tolerates the NSDAP party rise while closing your eyes to all the clearly visible actions that follow their Brandreden (=German for "inflamatory speech").
You can totally say something I don't agree with and I will fight for your right to do it, but as soon as you step over the line and violate other peoples rights, you waived your right to be tolerated by me.