Stuff the results of the crawl into Hadoop, write some MapReduce jobs to process the data, and then serve the structured data back up with HBase?