Hacker News Reader

Comment by tpmx | Hacker News Reader

I'm really curious about how the creators of the Wayback machine are working to save modern (perhaps sometimes somewhat unnecessarily overcomplicated) web pages that are using SPA "techniques". Have they implemented a googlebot-like crawler that interprets javascript and spits out.. some predigested final DOM tree? Or.. do you record all web-page-initiated network traffic and just let it replay, sort of? Lot of interesting research opportunities here, btw.

This is where archival meets browser/web tech, in a kinda complicated way. I would hope that people from both of these backgrounds have been working on this stuff together. If not, please start soon.

View on HN

A crawler has two high level options: parse the page, or render the page.

Most of our parser-based crawling is done by Heritrix (crawler.archive.org) and most of our render-based crawling is done by a proxy-based recorder similar to what you theorize (https://github.com/internetarchive/brozzler).

Thanks for sharing. That lets me sleep a bit easier.

Custom archivers, in many cases. The ArchiveTeam are a part of that (they're independent of IA, but work closely with them).

Feedback to info@archive.org also generally nets responses, though you may want to give that a week or so for them to get back to you.

I think this is a challenge that the archive.org/wayback machine people need to tackle to keep the general web page indexing working, going forward.

To me it's kind of obvious that "custom archivers" approach isn't going to scale to the entire web. "Customer archivers" implies some set of rules per "site engine". The problem is that that are probably (guessing here) 1M+ unique "site engines", or whatever you call them. I.e. the web is way too diverse.

There probably needs to be something generic that works with complicated pages 95% of the time, without previous knowledge of the API structure/dynamic.

If "custom archivers" is the plan to tackle the archival of the web for the past five and the next fifteen years.. I really think we need to do better.

Disclaimer: I'm not directly involved in any of this, though I've followed some general discussions.

1. The Web is much less diverse than you think it is, for two reasons.

2. Zipf/Power functions: the largest sites are a huge part of the Web. If people are spending (say) an hour online per day, and 40 minutes of that are FB/Insta, then all the rest of the Web gets 20 minutes. And much of that will be the next most popular site. Attention is fundamentally limited. In terms of experience, the Web is fairly small.

3. Which of course isn't content, but here again, much of the original content only hits a relatively small number of sites. I have looked into news site stats, and what you'll see for national papers of record (NYT, WSJ, WashPo, etc.) is about 150-500 items per day (Sundays are highest). The news services (AP, UPI, AFP, Reuters) tend to run 1,000 - 5,000 items/day. User-generated content sites are obviously higher than this, though much of that content is derivative (sourced/linked elsewhere).

4. Web engines. Even with a diversity of sites, there are a limited number of systems driving this. I've been noting a consolidation around news sites of a number of standard templates, and of course FB, YT, Instagram, Reddit, etc., will get you a bunch of specific Web engines. Which means it's less "one site per archiver" and more "what sites does this archiver work on?". And worse: over what period of time -- just because an archiver works on www.example.com from 2018-1-1 to 2018-12-31 doesn't mean it works from 2019-1-1 onward.

There are a few other approaches. Archive.today (archive.is / archive.fo) basically translate pages to their own representation. There are standardised presentation engines based on Readability (now defunct as a site, though its parser remains), used by Mozilla's Pocket and other tools (InstaPaper, etc.).

5. I also suspect we may see at least some normalisation around HTML5 (or subsequent) elements. Which I'd really like to have happen. NY Times, for example, has abandoned <table> for its own custom, and horribly broken, table-presentation alternative crap.

6. Yes, this leaves one-off SPAs which may be hard to archive. I don't know what the plans are for that.

(Now to see what IA have had to say about any of this.)

Update: from an IA response in this thread, "brozzler":

https://github.com/internetarchive/brozzler

Reply on news.ycombinator.com