It should be conpulsory to leave comments when you downvote a question. I always ask people to explain in the comment why they downvote me, and I would get more downvotes and 0 comment.
There should be an auto block feature. If you have it on and they downvote you, they get blocked and can't see your posts anymore.
Second thought about gamification. A downvote should cost the person a point or two of their own karma.