They absolutely do have power over you, that's exactly the problem.

People who get "canceled" don't just get a nastygram that they can throw in the trash. They are typically banned from speaking, fired / put on probation / "contract not renewed", their reputation is both privately and publicly tarnished such that #1 google result for your name will be that you were banned for CoC violation, damaging your career prospects etc.

Sure some people's lives are bullshit-proof but for many others witch hunts like this are nothing to laugh about.