HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by xena | Hacker News Reader
Full thread
xena
·
Do you respect robots.txt? How can I block your crawlers?
View on HN
skylerwiernik
·
Do you find this to be worse than googlebot somehow?
Reply on news.ycombinator.com