I thought about this quite a bit. I don't think I want downvoting. And I don't want effectless upvoting. The way it is right now is that you can add a paper to your library, and it will then feed into your personal SVM as a positive example of papers you like to see more of. Adding paper to your library as a type of paper you like to see more of effectively counts as an "up vote", and is what is sorted by when you go to the "top" tab.
Does that mean papers are "downvoted" by default, i.e. added to the negative example list?