I think a lot of people don't realize how discouraging it is to receive downvotes. They're handed out like candy for any perceived flaws in a question. I'd rather have a dialog in the comments to get the questioner to fix the flaws.
There should be an auto block feature. If you have it on and they downvote you, they get blocked and can't see your posts anymore.
Second thought about gamification. A downvote should cost the person a point or two of their own karma.