ParentFull threadtitanix2·As for open-source crawler goes, I've heard of SpiderLing (http://corpus.tools/wiki/SpiderLing) which is somewhat specialized for building text corpus.View on HN