Hacker News Reader

Comment by perlgeek | Hacker News Reader

I just tried the digg.com/robots.txt, and for it me it says:

User-agent: * Disallow:

Which explains everything without conspiracy theories

A "Disallow:" (without anything after it) is saying allow all robots access to the entire server. (If you need something beyond my word on the topic, there's a thorough explanation at http://www.robotstxt.org/robotstxt.html)

Yes, this seems to explain it.

http://www.robotstxt.org/robotstxt.html

No, it doesn't. Your linked example contains:

  User-agent: *
  Disallow: /

While the (new) digg.com/robots.txt contains:

  User-agent: *
  Disallow:

Those are very different. The former essentially disallows bots from crawling the entire site, while the latter disallows nothing - effectively allowing everything. The syntax is unusual, granted, for historical reasons.

Reply on news.ycombinator.com