Hacker News Reader

Comment by Yawrehto | Hacker News Reader

I hope this doesn't bankrupt the Internet Archive (either the legal fees or the case - I don't recall what they're asking for). It would be bad if the Wayback Machine, the biggest internet archivist around, went under, and also all the books, software, et cetera that the Archive hosts. I wonder if there's any way to archive all of the Wayback Machine (82.3 petabytes), or, better yet, all of the Internet Archive (which is, by my count, around 120 petabytes?) Who would have the capability/interest in doing so, ideally without charge?

View on HN

Shutting down IA altogether seems unlikely at this point (even if legal fees are substantial). They've written before about what the decision means (this appellate court affirmed what the district court did last year) and it doesn't touch things like the Wayback Machine: https://blog.archive.org/2023/08/17/what-the-hachette-v-inte...

Keep a close eye on the inevitable lawsuits that they're going to be up against. Obviously the music publishers are currently trying to kill the Great 78 Project and I expect other industries to pounce. (Especially the gaming and media industries). I can't see the Internet Archive exist in 2-5 years and we need to prepare for the worst case scenario. https://old.reddit.com/r/DataHoarder/comments/1bswhdj/if_the...

Discussion: https://news.ycombinator.com/item?id=39908676

Projects such as Anna's Archive are going to be much more critical than ever.

> Projects such as Anna's Archive are going to be much more critical than ever.

I look forward to all the new AA mirrors + contributions that are going to come from disgruntled archivists.

And everybody in the libraries/archives space going on downloading sprees. Most of us have personal illegal libraries. We just don't share them.

For wayback machine, are those compressed, deduplicated numbers? A semi-popular domain can have millions of results on their CDX api, but with https/https duplicated and about 90% of results are error pages or pages with deliberate garbage / LFI attempts in them.

Deduplication is not trivial. Each scrape is stored in a WARC archive, so you would have to unpack several large files, dedupe, and then pack them back up again. I believe they are at least compressed within each snapshot though.

Yes, that seems to be a silly way to go about it if your goal is to store the whole web and not just a single scrape. Of course anything that deduplicates data is more vulnerable to data corruption (or at least corruption can have wider consequences) so it's not a trivial problem but you'd think deduplicating identical resources would be something added the first time they came close to their storage limits.

We need to archive the archive.

Reply on news.ycombinator.com