What about downvoting incurring a (small) karma penality on the downvoter? I'd like to see such a penalty extended to flags as well, potentially.
[added] I've noticed that sometimes in a deep conversation between only two commenters -- where they cannot downvote each other -- one of them gets their comments downvoted within a minute of posting or so, suggesting that there are parallel accounts used to downvote as a method of intimidation or retaliation.
Maybe we need something like keeping track of the IP address of downvotes, and seeing if multiple downvotes to the same comment come from the same IP address? Or seeing if two accounts have a strong correlation of downvotes?