If the world around HN (including its community) changes, stasis can damage or kill it as well.
Specifically regarding the issue of the original posting:
- HN is already an important data source for large language model training. [1]
- To the best of my knowledge there is no freely downloadable and current data dump of HN. [2]
- The HN-API does not offer all the data that scraping can get. For example, if a post had ever hit the front page or the highest front page position reached, is an interesting data point that is missing.
- The Algolia-HN-API has the same limitations.
In my opinion this will lead to increased usage of the API and increased scraping which all costs money. HN might be forced to find a solution for this.
[1] For example, the RefinedWeb paper lists HN as one of only 12 websites that were excluded. From what I understand, it was excluded because it went into the final dataset unvetted. RefinedWeb was used for the Falcon model.
https://arxiv.org/pdf/2306.01116.pdf
[2] The closest thing is probably the Google BigQuery "bigquery-public-data.hacker_news" dataset. It claims to be updated daily, but really is from late September 2022. Also I could not find the download link which other data sets offered on BigQuery have. Does anyone know if I can download the complete thing anyhow?
https://console.cloud.google.com/bigquery?p=bigquery-public-...