The simple solution to that would be to require a comment if you are going to down vote. If you down vote you have to comment and say why.

Maybe that should be added as a feature for up vote to. To up vote or down vote you would have to comment. This would ensure that only people who were really serious about their opinion would vote at all.