Yes, I think you are correct. Downvoters have to be people with otherwise insightful comments (they need certain amount of karma to be able to downvote). Newcomers cannot downvote.

On a side note: I think that too many people misunderstand the purpose of voting. If you agree, upvote. If you disagree, do nothing. Downvote only if a comment is trollish or completely off-topic. If you think that the commenter doesn't understand the topic, help him overcome his ignorance, don't downvote him.