Hacker News Reader

Is there a technical reason that other crawlers can't just follow the googlebot rules? It's great that bots seem to obey the wishes of site owners, but I'm genuinely surprised that the yandex bot doesn't just follow the Google bot rule and ignore the general disallow rule.

View on HN

No technical reason. Following robots.txt is purely self-enforced, crawlers don't even have to read it if they don't want to. It's just part of being a good netizen.

Thanks. That's what I suspected - I'm just surprised that yandex is behaving like a good netizen :)

Reply on news.ycombinator.com