HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by davsti4 | Hacker News Reader
Parent
Full thread
davsti4
·
robots.txt isn't the law
View on HN
recursive
·
I doubt it would help any even if it was.
But in general being an asshole is not a crime.
AlienRobot
·
But if you don't follow robots.txt, Santa won't read your letter.
Reply on news.ycombinator.com