https://github.com/scrapinghub/splash
People often use Scrapy + Splash together in the Python community for crawling more dynamic websites.
A team I collaborated with is also working on a project to make Scrapy usable in a "cluster context", it's called scrapy-cluster. The idea is scrapy workers running across machines and a single crawling queue (in the current prototype, powered by Redis) in between them all.