Celery might be an option for distributing crawling jobs.