I have the same feeling about HN. I've been regularly visiting HN for a year now, commenting when I have something insightful and voting articles and comments up if they're exceptional. Yet I still don't have enough karma to downvote. The feeling I get is- I'm a long-time member of the community, what do I need to do to be trusted with a simple downvote?
This isn't that bad since I can still be part of any discussion, but it makes feel unwelcome and a little frustrated when I see something that should be downvoted.