HN has the same problem with upvotes. It would be healthier not to display karma scores at all, even privately. Use them for ranking etc but hide the numbers completely.
Yet at least some of them would gladly downvote a poster who behaves badly, e.g. is deliberately disruptive.