Analyzing One Million Robots.txt (That Apple and Google Won't Respect) | Hacker News Reader