Actually, you can't unless you meet a karma threshold. I have been on this site for 6 years and I still haven't reached it because my opinions are too controversial (apparently).
The comments you upvote/downvote are used to calculate your "opinion similarity" to the owners. Then your upvote/downvotes are modulated by that similarity. It's a way of using other humans to amplify the opinions of the owners -- they don't have time to read all the comments, so instead they filter out those who think the same way they do and let them do it.
As someone who likely has many different opinions from people from those of people at Y Combinator, this seems wrong.