I have pushed quite a few more updates to this since this post: Added Wikipedia snippets Refined the algorithm Changed how stuff gets pulled and crawled.