Python 3's asyncio and aiohttp do the job for me -- I can crawl several times as fast as the article's 111 qps with just a single process. https://github.com/cocrawler/cocrawler

Of course, it matters what you're doing with the page content, and how you're managing your metadata, and all that.