Hacker News Reader

Comment by enjo | Hacker News Reader

How can they truly 'de-list'? If the content is still open and accessible, what's to stop a spider from indexing it? I'm not a lawyer, but I have to imagine that falls well within the domain of fair use as long as your not presenting the content back for public consumption.

View on HN

Even if the spider doesn't respect robots.txt, the act of indexing is really obvious -- your webserver does not have to respond to their HTTP Requests if you don't want it to.

It's not as facile as 'lock your doors' -- instead just not answering when they knock.

> what's to stop a spider from indexing it?

a robots.txt that forbids a particular spider from accessing the content.

This assumes that the spider does respect the robots.txt, which googlebot does.

Is there a law against it? I thought robots.txt is more of a convention. However, Google already said that they would heed the robots.txt.

> Is there a law against it?

I don't know. I'd imagine that there isn't a law specifically written for robots.txt and scraping.

If a case went to court I'd imagine it would be settled based on laws meant for things like "keep out" and "No trespassing" signs:)

I think not heeding robots.txt would cause them more problems than it would solve.

Currently, it gets used for more than "deny/allow all" - it is selective. If I have to worry about stuff I don't want indexed being indexed, then I'm going to block all crawling. Yes, I know that wouldn't be easy, but I wouldn't be alone in my sentiment.

Even if this was proven as fair use, it would be quite difficult to make a usable search engine that didn't present at least some of the content back for public consumption.

Imagine if Google search results just had a bunch of links that said, "link 1", "link 2", etc instead of the title and summary of the page.

The idea you could index a site against its wishes remains to be tested; the custom so far has been not to do so. Google tried indexing books without rightsholder permission, then backed down.

If NewsCorp, NYTimesCo, WashingtonPost, ComcastNBC, etc. are unified against the idea of indexing-as-fair-use, and if (for business strategic reasons), the deep-pocketed MSFT and Google decide they can live with paying-for-permission (as a useful barrier to upstarts), who will prove the case in the courts or legislature?

Reply on news.ycombinator.com