I agree with you entirely except for one thing,
Downvoting without comment can do far more damage to conversation than can upvoting.
Imagine a real life conversation where you are speaking to another party, as you speak your mind, the other party continuously nods their head yes -- an upvote. You don't particularly feel the need to stop and ask "why do you agree with me?"
While on the other hand if, in the same situation, the other party started nodding their head no - a downvote, you would probably stop and ask them "why do you disagree with me?"
The conventions of this site essentially say that:
a) You can shake your head no, and the speaking party cannot ask why you disagree with them.
b) If the party asks why they are being met with disagreement, the response is the real world equivalent of a beating.
c) If you compare their negative, comment-less disagreement with any other similar situation, the response is a beating.
The result is that, after a couple exchanges like this, a typical person will not bother to speak their mind at all since they may end up with the online equivalent of a beating. Thus we end up with people that don't want to suffer the consequences of speaking out.
For example: http://news.ycombinator.com/item?id=1014800
Upvotes are like cake, it encourages people to work for more cake. But people will similarly work to avoid a beating, and non-participation is the easiest path to do so.