Not actively pushing back against something doesn't make one "part of the problem". It makes one a bystander, and no, bystanders are not part part of the problem they are neutrals. They could be more "helpful" for our cause if they where not, but they are not part of the problem, they are "untapped potential" if you will.
This kind of rhetoric bullies people into action and even tough I agree that we should push back in this case, this kind of bullying is not okay.
“ Let not any one pacify his conscience by the delusion that he can do no harm if he takes no part, and forms no opinion. Bad men need nothing more to compass their ends, than that good men should look on and do nothing. He is not a good man who, without a protest, allows wrong to be committed in his name, and with the means which he helps to supply, because he will not trouble himself to use his mind on the subject.”
So what? Why can't we acknowledge neutrality because some other people don't?
Obviously I'm not saying you always have to risk your job/career/livelihood/reputation to take unpopular (but right) action. That's just a nice-to-have.