Well, do you have something like this?:
http://setiathome.berkeley.edu/top_hosts.php
BTW, how do you ensure somebody is not submitting a malware (intentional or not) that can DDoS someone by crawling the same host at pretty much the same time?
http://setiathome.berkeley.edu/top_hosts.php
BTW, how do you ensure somebody is not submitting a malware (intentional or not) that can DDoS someone by crawling the same host at pretty much the same time?
80legs enforces throttles and rate limits on the sites that are crawled. We try to make a best guess at a rate limit for each site. Sometimes that best guess is too fast, so we take corrective action when we're notified by the webmaster. Users can't override the crawling rates.