If all scrapers did what you did, I'd curse a lot less at $work. Kudos for that.
Re 2 and 3: do you parse/respect the "Crawl-delay" robots.txt directive, and do you ensure that works properly across your fleet of crawlers?
Re 2 and 3: do you parse/respect the "Crawl-delay" robots.txt directive, and do you ensure that works properly across your fleet of crawlers?
However I do parse and respect the "crawl-delay" now, thanks for pointing it out!