Hacker News Reader

> this would be like a crawler ignoring robots.txt

Google ignores the noindex directive in robots.txt now. You're supposed to put it in your HTTP response headers or HTML meta tags...

View on HN

`noindex` was a Google-specific rule that was never officially documented nor supported. I think they were perfectly entitled to withdraw support for it, especially considering there are alternatives.

https://developers.google.com/search/blog/2019/07/a-note-on-...

Reply on news.ycombinator.com