Ruby & Nokogiri gem. You can also use Anemone gem to crawl the complete website.