Hacker News Reader

I noticed the Hacker News dataset that was published to big query was now a year out of date.

I have created an updated copy and made it available for download.

(This is the last 10MM entries, I can add the rest if people are interested.)

Thanks for doing this Aaron! Something to consider for the future: if you use grab-site (https://github.com/ludios/grab-site), which is a close cousin of ArchiveTeam's ArchiveBot, it'll retrieve all of the content from HackerNews (making the requisite http request and receiving the response for each URL, all of which is stored in WARC/CDX files), and store it in WARC/CDX files which can be uploaded to the Internet Archive. Those can then be integrated into the Wayback Machine by an IA admin at a later date. Something to keep in mind!

Thanks - was not aware of grab-site, I'll check it out!

Reply on news.ycombinator.com