Agree/disagree would only be tallied inline at the comment for reference -- there's no persistent reward for simply saying things many people agree with, nor penalty for saying unpopular things.
Then, it's OK for downvotes to serve the role this "more flags" idea does. Downvotes then unambiguously mean: uncivil; frivolous; factually wrong; repetitive; unwanted. (And, moderators could focus on highly-downvoted comments as much as 'flags'.)
(Previous comment, with more backlinks, on this idea: http://news.ycombinator.com/item?id=721853 )
Another related idea: never show a comment at 'maximum negative' score to a person who hasn't yet voted. Show it as '-3' instead (no matter how many net downvotes it's received). Then, there's always a motivation for adding your own independent judgement. (Once they vote, the true-but-truncated-to-range score can be shown.)
The poster could see the 'true' score, so they know if they've really touched a nerve.