I would like downvoting to require an explanation. It could be a single-line text box with the label "Explain why".
If you're serious about downvoting, it's no trouble to type in a short explanation such as "incorrect", "not funny", "obvious troll", "unnecessarily rude". (Those seem to be some typical reasons for downvotes.)
And if you're not quite sure why you're downvoting, having to explain would certainly serve as a determent.