Hacker News Reader

The reason shown in the crawl error is "Robots.txt fetch".

This isn't very surprising as 'fetch as Google' returns a page unavailable for everything.

(It's my understanding that a 404 on robots.txt is OK)

In the last few days I've seen errors with our robots.txt on sites that have NEVER had a problem with it and no errors. I even looked around on Twitter thinking it was a Google Bot bug. Then it just went away. Out of curiosity, why would a 404 on a robots.txt be OK? Shouldn't you want "Fetch as Google" to return the proper HTTP/1.1 200 OK?

I don't have a robots.txt, so -for this url alone- I'm expecting a 404. What I've read in the google documents is that when a request to get the robots.txt returns a 404, it will consider it as OK to crawl.

Reply on news.ycombinator.com