If you're looking to extract data from HN, use HNSearch. http://www.hnsearch.com/
There's no reason to scrape data from HN itself.
HTML APIs to come with the added benefit of being less prone to change, which I realize goes against conventional wisdom, but seems to hold in practice.
HTML APIs to come with the added benefit of being less prone to change
Really? I don't think that does hold in a lot of cases. Using HN as an example, it broke a year or so ago when PG changed how the job postings were listed. Again, the quality of an API can vary, but at least you would know what changed in that case.
Worthwhile communities need to be maintained by barriers. The kind of barrier that I personally enjoy most is, "Crappy enough interface that only people who really like the content will be motivated to show up."
I grant the existence of useful content there and a good API. However after one too many brushes with how differently moderators think about stuff than the people who I liked talking to did, I have demonstrated the truth of my comment by choosing to leave.
StackOverflow works for a lot of people. But for me, personally, a stripped down interface and less moderation is a better experience.
Presentation, along with content, is a better experience than just having good content.
You having the interface you want shouldn't impact my having the interface I want.
An API allows everyone to have the interface they want.