Downvoting gets unlocked at 501 Karma. Short of that, you have the ability to flag submissions where it makes sense to do so.
Submissions can never be downvoted.
Both submissions and comments/replies can be flagged, though I'm not sure if there's a karma limit on that facility.