Ahhhh, I bet you were using an iPad or iPhone or some other touch screen device, yes?
This is an issue: HN up/down votes + touch screen devices.
Yes, some will say be more careful, but I say up/down voting on sites in general are due for an interface/interaction redesign in light of ever increasing touch devices.