I've emailed pg, hopefully we can get back up and running properly.
pg, if you see this, I've updated my crawler to be fully compliant with robots.txt (my delay between requests was too low, even if the overall req/min was fewer than 2).
pg, if you see this, I've updated my crawler to be fully compliant with robots.txt (my delay between requests was too low, even if the overall req/min was fewer than 2).
No comments yet.