We should not remove down votes outright. I've seen a comment get downvoted to -4 because people didn't like the commentary, but then shoot back up to +10 when it was shown that they were correct after all.
The concept of "down" is conflated in people's minds in a way "up" isn't. It is used by people to mean either "I disagree" or "totally inappropriate". The first is not supposed to affect karma but the second is. If you simply replace "down" with "flag" it changes how people think about it.
As for accidental up-voting, moving the links so that there is separation would fix that.
I generally agree with that, but it reminds me of another problem. The Karma number is now an ever-growing and meaningless figure. I think it should be harder to improve your karma as you get higher, and it should be easier to lose it. That it actually has a meaning.
Inflation may be a problem some day, but for now I think it's under control. The leader board is still at a human understandable scale: http://news.ycombinator.com/leaders