I feel the same, actually. I'm a Marxist. I often get downvoted for mentioning something about socialism or social justice in general. I believe many of those downvotes are because the community here is not particularly fond of that ideology.
I get it. I'm not particularly fond of lots of ideologies. But I don't have tools here to differentiate between, "I disagree" and "Your comment is structurally bad (rude/breaks the rules/is low effort/etc."
I think we'd both benefit from breaking the coupling between those two reasons to downvote. (And yes, I'm aware that downvoting because you disagree is against the rules, but I don't think that stops anyone. :D )