That's not exactly true. Since more people can upvote than downvote, the threshold for downvote privileges is > 325 karma. So downvoting is reserved for those that frequently interact with the service in a positive manner. Downvoting could have had almost zero impact done to it due to the small set of people with that privilege.
As quantity increased, and newer less frequent users stopped upvoting, because there was no way to discern one's value in that interaction. Downvotes could have held at a similar frequency, because of it's smaller group. The decrease in upvoting could be accountable for a large portion.
Also, I rarely used a mobile device for interacting with HN, until 2-3 months ago, because the interactions are quite crappy (one of the reasons they mentioned for opening the API up.)
Edit: It's actually interesting, that the upvote/downvote ability wasn't considered impactful, since you mentioned it further down in the article.