What's a good tool/language to write scrapers in these days? A decade ago I was using ruby with mechanize and hpricot. I hope tools have improved since then, especially for scraping sites that use javascript.
From my work experience of working on a large scraping stack with thousands of integrations, I can say that we are very happy with our own custom framework, written in Go (https://github.com/PuerkitoBio/goquery for HTML parsing) and using headless Chrome for JS rendering.