I initially really liked this idea. I think the only problem is that it could lead to requiring more moderation. It's very hard to enforce quality. Right now I'm imagining a bandwagon chain of "You suck!" or other low quality comments just for the price of downvoting. I guess you could impose a minimum character requirement. At any rate, the idea of it being visible only to the user being downvoted is somewhat novel as well.