Hacker News Reader

Google at least claims that noindex will keep your site from getting crawled [1]. Do people think this is false?

[1] https://developers.google.com/search/docs/crawling-indexing/...

Strictly speaking no, that doesn’t prevent crawling - at the least Googlebot has to fetch the page to see the meta tag or the robots.txt to see what’s allowed, and it will periodically recheck for changes.

It doesn’t even prevent indexing. If a page is linked from elsewhere, Google will show it in search results even if noindex’d.

And why does Google get to set these rules on my site anyway? I didn’t agree to them.

Reply on news.ycombinator.com