http://digg.com/robots.txt <-- 404 Not Found
I know a robots.txt 404 shouldn't de-list you, but I would have expected digg to have one?
Maybe they requested de-listing and removed their robots.txt? Who knows!
I know a robots.txt 404 shouldn't de-list you, but I would have expected digg to have one?
Maybe they requested de-listing and removed their robots.txt? Who knows!
User-agent: *
Disallow: