pg has 147,000+ karma https://news.ycombinator.com/user?id=pg and the top of the leaderboard is tptacek with 159,000+ https://news.ycombinator.com/leaders
Rather than trying to break the habit you could always hire a ghost commenter to carry on. With you occasionally checking in to throw off everyone's scent.
I've also wondered about downvoting... I never see a button for that, but it seems like some people are still able to. Is it just the 'flag'?