Hacker News Reader

Comment by throwaway7767 | Hacker News Reader

One point to note here, is that the Wayback Machine obeys robots.txt retroactively, so sites can hide evidence by changing their robots.txt to disallow indexing of specific content. The data won't be purged from the servers, but it will not be displayed through the site.

This is quite disappointing (though probably wise from a legal standpoint). It makes it less useful if a party in a lawsuit can retroactively hide evidence of wrongdoing, and thereby deny access to evidence.

It's also a problem if a domain name changes ownership and the new owner suddenly adds a restrictive robots.txt - the old content will no longer be accessible even though the current owner has no claim to it.

View on HN

In 2009, there was an interesting case where a federal district court ordered a plaintiff to disable its robots.txt to allow the Wayback Machine to disclose an old version of the plaintiff's website.[1] Someone at the Internet Archive provided a declaration stating that it would place a significant burden on them[2] to respond to a subpoena themselves, whereas all the plaintiff had to do was modify its robots.txt. The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt.

[1]: http://www.american-justice.org/upload/page/123/69/docket-18...

[2]: https://tonybox.net/tmp/ia_decl_pacer.pdf (downloaded from PACER)

> The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt.

How is this not indirect self-incrimination?

For one, losing in a civil trial isn't the same as being subject to criminal penalties, so you can't self-incriminate unless what you're hiding would itself have been a crime - many contractual disputes have no criminal dimension whatsoever and so your liberty isn't at risk.

But more to the point, archive.org already has a copy of what you previously made public, and there's no reason to burden them with the expense of manually retrieving it as they're not party to the case, and it would be wasteful of the court's resources and archive.org's time to make them sue you for the expense involved.

Here's a more detail breakdown of how the 5th applies in a civil litigation context: http://www.litigationandtrial.com/2013/04/articles/attorney/...

The bottom line is that the 5th can help you stay out of prison, but you can't use it just to avoid losing a dispute.

Sounds like saying we already have clear evidence but it'll take time to dig through it so you are now compelled to show us where you hid the body.

The government has the right to use force in an investigation.

Things you have in your mind or - in some cases - in an extension of your mind (you're secretly actually guilty, you know where incriminating evidence is, or a safe combination or password that you haven't forgotten) cannot be compelled.

Things you possess (the contents of your home or gun safe, the thumb drive clenched in your fist, the robots.txt file on your server) can be compelled with a search warrant or subpoena. They are your things, and they could incriminate you, but that's not what the 5th Amendment is for.

Laconic.net is a resource I have enjoyed. Here is the 5th amendment passage: http://lawcomic.net/guide/?p=2470 or go back for a history lesson.

Whoops, that should be "lawcomic.net".

The modern Miranda passage starts here: http://lawcomic.net/guide/?p=2589

A subpeona was already issued and this information could have still been extracted from the Internet Archive directly, just at a higher technological hurdle. If the information was still on the plaintiff's servers they could have just extracted it from there as well (or any backups that they might have had laying around), since it was deleted they needed other ways to recover it.

robots.txt is assumed to only be used to block automated access to a website by search engines and other crawlers, not to hide it from human view as well (in fact, it makes sense to NOT use it to hide things from humans as it announces the presence of content that you are trying to hide). Removing a file that is only to be used to prevent crawlers from indexing certain parts of your site (because they are irrelevant for search results) is hardly self-incrimination (again, because you are advertising the fact that content exists at this location and you don't want it indexed).

It's available in the wayback machine, it was already made publicly available at a previous point in time. This is primarily reducing the courts burden.

Does self-incrimination apply to companies?

Thanks for answering a question I had been worried about for ages -- did changes to robot.txt make the internet archive delete it's history, or just hide it.

Thanks for making the PACER doc available -- is it in RECAP?

deny access to evidence

Subpoena?

That goes to weight more than admissibility.

The only thing that the judge ruled in this case is that it CAN be used.

Very interesting.

At least they cannot add content.

And it is easy to know if they may have hidden something. Since the robots.txt should be public. I hope. May be a judge can make them unhide. I don't know.

Reply on news.ycombinator.com