Ah, ok, that makes a lot more sense. I was imagining removing comments that had been downvoted; not removing the link to downvote.
I generally agree with that, but it reminds me of another problem. The Karma number is now an ever-growing and meaningless figure. I think it should be harder to improve your karma as you get higher, and it should be easier to lose it. That it actually has a meaning.