Humor is the gateway drug. If people start upvoting jokes, then HN will turn into Reddit. Then into Digg. Before you know it, you might as well call it HNchan.
I don't mind if people don't want to upvote a joke. That is fine by me. But, we don't need to downvote them either.
I disagree with jokes being the catalyst which will change HN into Reddit or Digg. I don't think it is the comments that turn the worm, rather the quality of the bookmarks which are submitted. If the heart is true, the body will be also.