You have access to the voting log of HN?

For the record, you can not downvote posts that reply to yours.

You could not know that because you do not have enough points yet to downvote at all.