Which is not to say that just adding complexity is the right fix. Sure, you can add dislike + disagree + ijusthateyourguts buttons, or force votes to be justified (funny/incorrect/disagree/...), or whatever. But it's not obvious what's worth the overhead (UX-wise), and if it ends up boiling down to just finding more nuanced reasons to toss things into one bucket or another (or even a single score), I'm not sure if we're any further ahead.
Perhaps clustering is a better model than polarized buckets? As in, this comment belongs to the set of things that left-leaning people seem to support. This other comment belongs to the conspiracy theory #1 cluster. (The system would likely not know or care what the clusters mean.) If you choose to engage with the idiots who love cluster A, you can do so, and if you want their noise out of your feed then you can do that too.
You'd still need to decide what input signals will enable this -- you could do it with just upvotes and correlating users' activity across a site, but you would need to evaluate that in terms of what best supports the model (clusters instead of buckets).
And clusters aren't the end-all be-all either. A racist, misogynistic comment that also says something that group X strongly agrees with isn't going to be fully described by being in the "jerk comment" cluster. But we have to deal with a low-dimensional approximation in order for any mechanism to be worthwhile.