The Scrapy library written in Python (https://scrapy.org) is excellent for writing and deploying scrapers.