Crawling a quarter billion webpages in 40 hours (2012)
michaelnielsen.org
michaelnielsen.org
You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this:
- SPAs with front end only rendering are hard to scrape with a traditional http library.
- Anti bot/scraping technology fingerprints browsers, looks at request patterns, and browser behaviour to try and detect and block bots.
Using a "real browser" is often advised as a way around these issues.
However from my experience you should avoid headless browser crawling until it is absolutely necessary, I have found:
- Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources.
- Most SPAs now have some level of server side rendering, and often that includes having a handy JSON in the returned document that contains the data you actually want.
- Advanced browser fingerprinting is vanishingly rare. At most I have seen detection of user agent strings and comparing them to http headers and the order of them. If you make your http lib look like a current browser you are %99.9 of the way there.
Sometimes the API will need some form of session tokens found in HTTP meta tags, form inputs, or cookies. Sometimes they have some CORS checks in place that are very easy to bypass from a server-side environment by spoofing the Origin header.
For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.
I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or that just contained a bunch of javascript with a headless browser.
If Google can crawl it, then you can too. And while Google doesn't use a headless browser (or at least I assume they don't) they absolutely do execute javascript before loading the content of the page. And they execute the click event handlers on every link/button and when we use "history.pushState()" to change the URL Google considers that a new page.
You're just going to get a loading spinner with no content if you do a dumb crawl (I disagree with that and think we should be running a headless browser server side to execute javascript and generate the initial page content for all our pages... but so far management hasn't prioritised that change... instead they just keep telling us to make our client side javascript run faster... imagine if there was no javascript to execute at all? At least none before first contentfull paint)
This is true for some SPAs, but not all SPAs, and there's not really any way of telling which is which.
I don't personally attempt to crawl SPAs because it's not the sort of content I want to index.
That being said, with sites like HN, Reddit, LinkedIn, Twitter, news outlets, etc. the lines between “document” and “application” get blurred. In some ways they’ve built a micro-application that hosts documents. Content can be user submitted in-browser. Content can be “engaged with” in browser. Some handle this blurring better than others. HN is an example IMO of getting it right where nearly everything that should be addressable (like comments) can be linked to. Others not so much.
(As an aside, I love marginalia!)
It's simply not feasible to treat them as any other website if you wanna index their data.
- The host has a tendency to block unknown user agents, or user agents that claim to be a browser but are not
- Anything that requires client side rendering
I'd suppose both problems are more pertinent in 2023 than they were in 2012.
At web scale, the issue appears on at what point would you be required to use a headless browser when not using one in the first place e.g. if React is included/referenced. Perhaps some simple fingerprinting of JS files would do, IMO in reality the line is very blurry, so you either do or don't.
[1] https://www.adriancooney.ie/blog/web-scraping-via-javascript...
(Previous comments about it from 2022 here: https://news.ycombinator.com/item?id=31205139)
For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need.
For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.
Instead of setting up some kind of partnership with our vendors, where they just send us information or provide an API, we scrape their websites.
The old version ran in a hour, using one thread on one machine. Downloaded PDF's and extracted the values.
The new version is Selenium based, uses 20 cores, 300GB of memory, and takes all night to run. It does the same thing, but from inside of a browser.
As a bonus, the 'web scrapers' were blamed for every performance issue we had for a long time.
How are they scaled?
If you need cookies/headers, you can always open the browser, log in and then make the same requests in the console, instead of waiting for the browser to load and scrape the UI (by xpath, etc.)
sounds weird going in circles: - SPA call some URL - SPA use the response data to populate the UI - You scrape the UI
instead of just calling the url inside the browser? am i missing something?
Changing your IP address with AT&T is a pain (even though the claim is that your IP is dynamic, in practice it hardly ever changes) so I opted to contact the CDN vendor by filling out a form and luckily my ban was lifted in a day or two. Nevertheless, it was annoying that suddenly a quarter of the websites I normally visit were not accessible to me since the CDN covers a large swath of the Internet.
The major content providers tend to go on a whitelist only based approach, you're either a human-like visitor or facing their anti-scraping methodologies.
Overall, it's not as bad as it seems. I doubt anyone would accidentally damage their IP reputation doing otherwise above-board stuff.
Every ISP just uses DHCP for router IPs. It's dynamic, you just have to let the lease time expire to renew it.
Or, have your own configurable router instead of the ISPs so that you can actually send a dhcp release command though they don't all support this part. Changing MAC Address will work otherwise.
We had a periodic script that had all of these caveats about checking telemetry on the affected systems before running it, and even when it was happy it took gobs of hardware and ran for over 30 minutes.
There were all sorts of mistakes about traffic shaping that made it very bursty, like batching versus rate limiting, so the settings were determined by trial and error, essentially based on the 95th percentile of worst case (which is to say occasionally you’d get unlucky and knock things over). It also had to gather data from three services to feed the fourth and it was very spammy about that as well.
I reworked the whole thing with actual rate limiting, some different async blocks to interleave traffic to different services, and some composite rate limiting so we would call service C no faster than Service D could retire requests.
At one point I cut the cluster core count by 70% and the run time down to 8 minutes. Around a 12x speed up. Doing exactly the same amount of work, but doing it smarter.
CDNs and SaaS companies are in a weird spot where typical spider etiquette falls down. Good spiders limit themselves to N simultaneous requests per domain, trying to balance their burden across the entire internet. But they are capable of M*N total simultaneous requests, and so if you have a narrow domain or get unlucky they can spider twenty of your sites at the same time. Depending on how your cluster works (ie, cache expiry) that may actually cause more stress on the cluster than just blowing up one Host at a time.
People can get quite grumpy about this behind closed doors, and punishing the miscreants definitely gets discussed.
[1] https://blog.adnansiddiqi.me/scraping-dynamic-websites-using...
Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.
My fastest implementation in python was actually using threads and was much faster than any async variant
Also kinda wish the author paid any sort of attention to the fact that doing this incorrectly may create a flood of DNS queries. At least have the decency to set up a bind cache or something.
The operating system does some DNS caching as well, but it's not really tuned for crawling, and as a result it's very easy to end up spamming innocent DNS servers with an insane amount of lookup requests.
If you're going to do way more DNS lookups than are expected from your connection, it's a good custom to provide your own caching layer that's scaled accordingly.
Not doing so probably won't break anything, but it risks degrading the DNS performance of other people using the same resolvers.
This is a massive simplification but most useful for a beginner.
Additionally, asyncio is not the same as multithreading, because typically asyncio is powered by a single-threaded event loop and use of a mechanism like select/kqueue/IOCP/epoll.
You can probably do it on a single core in a low level language like Rust/Java.
Assuming you're crawling at a civilized rate of about 1 request/second, you're only so many network hiccups away from consuming the entire ephemeral port range with connections in TIME_WAIT or CLOSE_WAIT.
It's very easy to end up with tens of thousands of not-quite-closed connections while crawling, even with SO_LINGER(0), proper closure, tweaking the TCP settings, and doing all things by the book.
It's a different situation from e.g. having a bunch of incoming connections to a node, the traffic patterns are not at all similar.
if they replicate it in 2023 it would be pretty interesting to me. i can think of a few times a year i need a good scraper.
but also thought it a good look into 2012 Michael Nielsen, and into thinking about performance.
There's also a list from Netcraft [3].
[2] https://www.domaintools.com/resources/blog/mirror-mirror-on-...