Hacker News Reader

Archive.org is one of the greatest sites in the world right now. The only negative thing that I have to say about it is the retroactive removal of sites that have a anti-bot stance on robots.txt (even if it does not ban archive.org specifically). I would understand if they did not crawl sites that had such a robots.txt but I find the retroactive removal a dangerous policy.

View on HN

It's unfortunate but they operate in a sufficiently grey area of law that they really need to bend over backwards to comply with the wishes of someone who owns or may own the content. I don't particularly like it either but I understand why they feel it's a prudent policy.

If internet access is a human right, which I believe the UN said it is? Well, then we could have special laws for internet archival? Much like we have laws that prevent the destruction of historical sites. It's not that anybody is making a quick buck at the expense of somebody else.

Reply on news.ycombinator.com