It doesn't matter. It didn't matter for all the people in East Germany the communists took away and tortured and often murdered. It certainly didn't matter in the Soviet Union, either. Your silence and self-censorship won't matter in the end. The Soviets often just targeted random people because it chilled dissent overall.

Silent or not, party member or not, you will be dragged from your home in the night and never seen again. This is the reality of what it was like.

So you might as well be brave and speak out against it because that's the only actual defense you have.