Re: crawling being too hard

Have you contributed your crawl data to common crawl?