Hacker News Reader

Comment by mikece | Hacker News Reader

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

View on HN

Don't know about large scales, but just today I threw together a script using selenium, imported pandas to mangle the scraped data and quickly exported to json. For quick and dirty, possibly one-off jobs like that, Python is a great choice.

The Scrapy library written in Python (https://scrapy.org) is excellent for writing and deploying scrapers.

I think Python makes sense, at least for the prototyping phase. There's a lot of trial and error involved, and Python is quick to write.

I find javascript (node) to be best suited to web scraping personally. Using the same language to scrape/process as you use to develop those interfaces seems most natural.

Especially with stuff like Puppeteer which allows you to execute JS in context of the browser (which admittedly can lead to weird bugs as the functions are serialized and lose context)

I'm using this exact strategy to scrape content directly from DOM using APIs like document.querySelectorAll. You can use the same code in both headless browser clients like Puppeteer or Playwright and DOM clients like cheerio or jsdom (assuming you have a wrapper over document API). Depending on the way a web page was fetched (opened in a browser tab or fetched via nodejs http/https requests), ExtractHtmlContentPlugin, ExtractUrlsPlugin use different DOM wrappers (native, cheerio, jsdom) to scrape the content.

[1] https://github.com/get-set-fetch/scraper

I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.

I'd say that really depends on your scale and what you're doing with the content you scrape.

In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.

Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.

I would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.

My experience was with e-commerce scraping. Not many domains, but a massive catalogue.

Reply on news.ycombinator.com