I've seen this article.Great article about distributed crawler. Mine just is a demo version of his.