Building a bot to crawl all of the Internet and save it to an index is a fairly straightforwards task. As Google and Pagerank proved though, it's the algorithm you use to search that index that's valuable. Any idiot can try and run grep against said index and give 30,000 results, of which the one you want is on page 53. So writing the crawler to build the index isn't really a competitive advantage.

Why then reinvent the wheel and spend untold amount of resources re-crawling the web, when Bing will let you use theirs? What secret sauce for crawling web pages does doing your own crawl bring to the table?