I’ve had the downvoting spree happen to me a couple of times. But I don’t agree with you because preventing downvotes/flagging, or identifying users who appear to downvote in bad faith, will just spiral into worse behavior. HN isn’t perfect but it’s better than the alternatives.
My HN karma doesn’t mean anything. Losing a few points now and then to a person with too much time on their hands doesn’t bother me.