Looks pretty interesting to do async on python. I once did something similar in node (async by default) with a few lines of code. I think I scraped 12 or 20 million real URLs in 8 hours on a $5 cloud VM. It was limited by network bandwidth.