My view right from the start of the internet has been to make quarrelling exact a cost, just as it does in real life. Thus downvoting should cost equal karma...
Yet at least some of them would gladly downvote a poster who behaves badly, e.g. is deliberately disruptive.