Hacker News Reader

> To be fair, Google respects robots.txt

Not all of their crawlers respect robots.txt

Do you have an example? A crawler ignoring robots.txt sounds like a bug, and if you have details I can file a bug with the appropriate team.

FeedFetcher User Agent