…but if we remove downvotes, they will just stay at 1 instead of getting downvoted. Later they will still be able to shoot up. They will only get flagged if they are actively in the wrong spirit of the community. (spam, abuse, etc..)

The concept of "down" is conflated in people's minds in a way "up" isn't. It is used by people to mean either "I disagree" or "totally inappropriate". The first is not supposed to affect karma but the second is. If you simply replace "down" with "flag" it changes how people think about it.

As for accidental up-voting, moving the links so that there is separation would fix that.