I've contacted HN before and they've told me they auto-downvote certain kinds of content. It's not a stretch to assume their views are influencing that criteria.
Oh yeah, I think I remember reading they demote submissions that they think will cause low quality discussions. I don't think that's related to downvoting or your karma.