A few months ago they increased the # of karma required before you could downvote. I've noticed things have been much better since.
I've found that most of the people with high karma and low averages just say snarky one-liners all of the time and some of that happens to land with people (for better or worse) giving them a decent amount of karma despite a lack of contribution.