Analyzing One Million Robots.txt (That Apple and Google Won't Respect)intoli.com2 points·testcross··0 commentsOpen articleSaveView on HN