Or people who get the joke are downvoting it because they like serious discussion and don't want this place to eventually be filled with cheap one-line jokes like Reddit.
I've noticed a lot of people just downvote his comments without even trying to understand anything he says - in this case, it was an amusing Earthbound related joke, and unsurprisingly downvotes follow because of the commenter, and not the comment itself.