IMO, the most important work that upvotes and downvotes do is to move comments up and down the page.

In that context, HN consensus on downvoting jokes makes more sense. I like a good joke, but they belong on the bottom of the page after the "serious" conversation. It's unfortunate that it's overloaded with karma, but complicating the upvote/downvote system has other drawbacks.