Note how you have just did exactly what you criticized ( “silence legitimate criticism of others” by dismissing those that called out cancel culture)
In past times, nonviolent forms of 'cancelling' that would seem familiar to us today were also employed. It was/is called 'shunning.' Our modern term is just a euphemism for that.