There's an extremely vocal and downvote happy minority of users on HN that try to force everyone into agreement or cowardice. They will change language definitions and weaponize them against you, or try to turn you into some sort of monster for having differing opinions. Basically anything to silence opposing opinions.
There is also an extremely quiet majority of users on HN that do not wade into those conversations out of what I presume is fear of the brigade or just being exhausted from it all... but they will support your arguments via voting (the only tool at their disposal).
So... just wait out the extremists and allow the normal users to judge the quality of your arguments for what they are. Over a several hour or day long period, you will have a more accurate representation of what HN actually thinks about your arguments. It will be a roller-coaster of ups and downs along the way - that's normal.
Stop caring about the points and start caring about the quality of your arguments or unique perspectives you can communicate. Stop downvoting simply because you disagree, and start downvoting for poorly crafted, provably incorrect and/or poorly communicated arguments.
Speak your mind, no matter what it says. HN should be a place for open discussion from people of all walks of life unified by a loose commonality of a love for technology. It should not be a cesspool of unidirectional thoughts - and those who remain quiet contribute to that just as much as the brigade.
There is value in having pure drivel or outright false/hateful information hidden from view. It serves no purpose.
This is the age old problem of every site with a social aspect though. I don't blame HN.
I don't 'care about' it as a score, but I do pay attention to it as an indicator that maybe there's some reply/discussion going on and I should check /threads. It's also a slightly scary proxy for time sunk into HN, I suppose!