Well, it already has a leaderboard and some other sorts of rankings, http://news.ycombinator.com/lists which I find very useful. Not sure why the link to it is buried at the bottom of the page.
Of course, hiding/obscuring karma could remove some of the motivation to upvote comments. It would be interesting to see how usage patterns changed if it was hidden/obscured ...
An option to hide karma would be useful as well, for people who aren't interested (like you). I wonder if there's an extension for some browser which does that; if not, it probably wouldn't be too hard to write one [stripping out everything in the form of "# points" and the (#) next to the username in the bar at the top of a page ... adding display:none to all the score_# classes and applying a bit of filtering to pagetop would do that.]
I'm also in favor of getting rid of downvotes, mostly because I see them abused too much (people being downvoted for expressing unpopular opinions, especially anti-apple), but that's neither here nor there.