Hacker News Reader

Comment by danso | Hacker News Reader

I think a case could be made that the Archive should be the reference by default for citations -- that is, when a URL is cited, the Archive's snapshot API is triggered, and the archive URL is used as the direct link with the original URL being the "backup". Sites/pages change so much that it's more helpful to point the average user to what a page looked like when the reference was accessed and included. It may rob the original URL some traffic, but Wikipedia isn't meant to be a link-aggregator for external content.

The obvious roadblocks are:

- Websites, even content-focused ones, can be so complicated in terms of JS that the Archive might not capture it accurately.

- The Archive's policy is to honor robots.txt and other no-archive directives.

View on HN

It may rob the original URL some traffic,

That is an advantage. It reduces extraneous incentives to post links, which should be only about the information they provide and not about page views. At wikipedia's scale, and sensitivity to information purity, that's relevant.

See also the rel="nofollow" decision from a few years back.

It'd be a real shame if publishers decide to opt out of the archive to reclaim "lost" pageviews.

It's probably good that it's possible to opt-out of being archived but what annoys me is a useful domain changing hands and the new owners deciding to wipe its history.

Surely that's not possible?? And there is no way to find the old archive, even under an alias?

It's not wiped (they don't delete anything) but if robots.txt forbids indexing then access to old archives is blocked. It can be quite aggravating, but at least the information is not lost---if robots.txt or archive.org's policy changes, it can be retrieved again.

> - The Archive's policy is to honor robots.txt and other no-archive directives.

What bothers me is that they do so retroactively based on the current robots.txt, not the one contemporary with the archived content. So if a domain parker takes over a domain, and their robots.txt excludes everyone from every page (or everyone but Google), then archive.org no longer provides its archive of the old content.

That seems like an awful bug, is this by design?!

By design from the last time I saw it discussed. The idea is that a change there could indicate that there was a mistake and the data shouldn't have been crawled for one reason or another. There's just no way to know in an automated fashion.

I'm inclined to say that robots.txt should be ignored altogether and removal requests should be handled per individual case. But I guess that goes too far...

Still, if the domain owner changes, they should not be able to remove content from old archives. That's like being able to remove stuff from encyclopedias about a palace somewhere, just because you live in the place where the palace once stood.

They are The Internet Archive after all, it's logical to archive contents like the domain owner and robots.txt for a given point in time. A change of owner can be easily detected.

> removal requests should be handled per individual case

Are you going to fund the internet archive to handle that workload?

> if the domain owner changes, they should not be able to remove content from old archives.

Why not? If that work belongs to anyone, it is the current domain owner. Why does the fact that the internet archive happened to crawl it mean that suddenly they lose control of their information?

This is elevating the Internet Archive from 'hey it's cool someone made a copy of that while it was up and no one cared' 'because the Internet Archive crawled it the world has absolute rights to that information from now on, wishes of the owner be damned.'

> If that work belongs to anyone, it is the current domain owner.

Not at all. A domain parker taking over a domain does not imply they have any rights over all the content that the previous owner of the domain posted.

Outside of any other information, it does imply that.

> Are you going to fund the internet archive to handle that workload?

I figured someone would ask that. I don't have an immediate answer but it's a good question (upvote for that). My hopes are just that removal requests are not too frequent. But without current numbers (of number of pages hidden after-the-fact and current removal requests) this is guesswork.

They could charge for it perhaps? A dollar per request. Doesn't seem too unreasonable for something you mistakenly made available to the planet. It doesn't have to be per page, so if you made a million documents available all under example.com/hidden/ then hiding that folder is a simple action and costs just one dollar. You're paying them for their time.

In the Netherlands, if you want your personal information (e.g. phone number; email address) removed from a company's systems, you can request that and they must grant it if they have no reason to keep the data any longer. And you can make requests to see your data, etc. But the law allows for companies to charge for this and I've seen example amounts (I think around 3 euros) somewhere. It's a somewhat similar situation.

So I don't have a single good answer, but I think by-case is a better way to go (and worth thinking about, at least) than just using the current approach.

Yes, it's by design. It's to stave off the inevitable flood of legal threats and takedown demands.

If someone requests content be taken off, they instruct them to update their robots.txt. The content is not removed but will not be shown through archive.org as long as the robots.txt exclusion is in place.

There was a court case where the plaintiff wanted to subpoena the Internet Archive for evidence (since the defendant had since blocked the content with robots.txt). They sent an expert to testify that complying with that kind of thing would be too much of a burden for them, and suggested that the court force the defendant to change their robots.txt. The court agreed.

I'm not sure there's a good way to work around this - the alternative is to only respect the robots.txt as it was when the snapshot was taken, at which point once a confidential page is in the archive you can't (easily) remove it again.

Perhaps access to archived pages should only be blocked when the ia_archiver user agent specifically is denied in the robots.txt. That way they aren't inadvertently blocked by a generic robots.txt that denies everything (which sometimes occurs with parked domains), but there's still a way to deny the Wayback Machine if you really need to.

This, exactly. Or, even better, use a separate user-agent ("ia_retroactive" or similar) for retroactive removals.

Well, then everybody would claim the same right and you'd have to maintain full list of bots and keep it up to date. This doesn't sound scalable, * should still mean "everybody".

robots.txt only tells bots not to crawl the website. It doesn't say anything about indexing or archival of pages.

It would be perfectly possible to have the Wayback Machine respect the robots.txt and not have it crawl or archive any new pages, whilst making pages that have already been archived accessible unless a specific user agent has been denied.

I would say that the default behavior should be to respect the robots.txt of the time of the snapshot and only revert archival of accidentally cached pages that were never intended to be public.

Sadly, it requires a bit of human intervention there.

The internet archive wouldn't work if its crawler wasn't fully automated. You can't handle the whole internet in a way that requires human intervention.

The problem for manual deletion via human intervention is that if you do it outside of robots.txt you then need to ensure the identity of the owner, which makes it much more complicated and costly.

Maybe robots.txt could have clauses that say whether to apply entries retroactively. True, domain parkers could enable it, but I don't think most would, since it's extra work for no benefit - the point is usually not to erase history but to protect current site.

Since the number of pages referenced from Wikipedia is limited, the archiving could be also done by other parties. Maybe just setup a raspberry, xTB hard disk, download everything and then distribute over IPFS (or something else) (as backup in case Archive.org removes the pages).

To take into account Javascript etc, one could also capture a png snapshot of the page.

Wait, so every time someone makes a citation on wikipedia it causes Archive.org to archive that URL? This seems like it could be abused by a malicious agent, no?

You can already just tell archive.org to archive a URL. How would this be different?

You could have a "reverse DDoS" where archive.org blocks requests from Wikipedia, removing the ability to add Wikipedia citations.

There's no such thing as a "reverse DDoS", that's just a regular denial of service attack.

A theoretical reverse DDoS: You somehow prevent traffic from reaching their load balancer so it automatically scales their AWS instances to zero.

Your minimum instances would always be 1 in the autoscaling group associated with your ELBs (or now, ALBs).

Regardless, neither the Internet Archive nor Wikimedia use AWS or other cloud providers, as it would be prohibitively expensive. They both run their own infrastructure/ops.

Mostly a joke, just trying to think of what that term would mean.

What I meant was:

1. Get Wikipedia to send lots of requests to Archive.

2. Archive blocks requests from Wikipedia.

3. Wikipedia citations are disabled.

In other words getting Wikipedia to DDoS Archive, so that Archive's defense hurts Wikipedia.

A very silly scenario of course, just coming up with one for why an attacker might want to indirectly DDoS Archive via Wikipedia.

That's still not a DDoS, as its missing the Distributed element. Almost by definition if you can block all requests at source its not a DDoS, its just a regular old DoS.

What would probably happen between 1 and 2 is that archive.org notices spike in traffic from Wikipedia, talks to Wikipedia engineers, they find out who is generating all these requests, and block them from editing Wikipedia.

Though it's certainly possible to generate enough junk edit traffic to cause disruption on Wikipedia, but that's nothing new. It's the nature of Wikipedia as the resource - it trusts the internet community to be good on average. So far it worked.

I think a reverse DoS attack would be one that actually increases the capacity of the service. I'm not sure what scenario would allow that to happen though.

Some streaming sites (e.g. 4 on Demand, IIRC) have a peer-to-peer element; you could reverse-DDOS those by running a lot of computers "seeding" (perhaps by leaving them on the page but at the end of the video) distributed all over the globe.

Well, in my hypothetical world, the Wikipedia editor page would include a widget in which a user could submit a URL, and Wikipedia would generate the markup, including an archive URL. Deciding how to implement this so that it works efficiently will have some overlap with issues on that 2nd hardest computer science problem of cache invalidation.

Good idea, after that it's really just rate limiting and perhaps only looking up the same URL every X days. Perhaps a per domain limit as well. No problem they probably haven't already solved.

Easy enough, but what do we call it?

The Wikiarchivator?

There's already the citation tool in the default editor. Shouldn't be too hard to just fire off an AJAX request to the Archive when the citation is added, right?

Reply on news.ycombinator.com