Now that a lot of my scraping needs require me to render the DOM, I have had to run a headless web browser (previously PhantomJS/Splash and now Chrome). Chrome released the Puppeteer library which is the most straight forward way to interact with Headless Chrome. This is in Node so that is what i’m using.
On my last project we needed to parse through a large amount of HTML files to extract relevant information (about 70,000,000) and put it in a DB for further analyses. I have done similar things in Python before, but there are blocking/concurrency issues that can make it slow and frustrating to build.
My solution was to just do this in Go. I can then run my extraction in a goroutine. Go’s selling point for me was speed (relative to Python) and dead simple concurrency - which made it the right tool for the job.
Edit: Spelling, iOS’s autocorrect can be so annoying sometimes :)