You need 501 karma to downvote a comment. I think the threshold is a feature, not a bug.
Try to avoid political threads. Each one has his/her opinion and wants to donwvote any alternative.
Technical threads are more safe. Sometime people get downvoted for being wrong, but I don't like it so I upvote gray comments (unless they are offensive or very wrong).
Anyway, I use the downvote very seldom, perhaps only once per day. What do you want to downvote?
The stories fall from the front page when they get many flags. You need only 31 karma to flag. Also when a story gets much more comments that upvotes, it gets a penalty. The mods can sink a story, or remove the penalty of the flags or the automatic penalty.