Great point, but I wasn't aware that there are "usual, legitimate" reasons for downvoting, and that these are well-known enough that you can call them the usual reasons. Seems like everybody just uses their best judgment.
That said, I didn't downvote him, I simply didn't vote. I don't feel a need to vote on everything, and that extends to downvoting.