How can they crawl the entire web ( or a big portion of it) everyday without spending tens of thousands of dollars in server / bandwidth costs?
I know there are lots of open source crawlers like Stormcrawler and all, but the cost of running it at web scale everyday is prohibitive isn’t it?