It's specifically and deliberately user hostile. People unwilling or unable to make the effort or take the time usually get frustrated and leave, or ask questions and get enculturated.
You're not expected to understand everything instantly, and the usage of these things is expected to change over time anyway. Their purpose wasn't set in stone to begin with, and already has changed. For example, the down-vote used to mean "This doesn't belong on HN" but has evolved to include "I don't agree".
And flag does more than just act as a downvote on submissions - it indicates that the item shouldn't be on HN and is used for spam, troll, seriously off-topic, and by some, for repeats of recently submitted stories. Enough flags and a submission is auto-killed.