How to Crawl the Web with Scrapy
babbling.fish
babbling.fish
I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go.
We ended up just using requests to crawl the web. Then post-process the web pages in the next step.
Many thanks to Zyte [1] (ex-ScrapingHub) for open-sourcing so many wonderful tools for us. I've spoke to Zyte's CEO, and was really fascinated how he still being a dev person while running such a big company.
[0] https://newscatcherapi.com/news-api [1] https://www.zyte.com/
I agree (but in my case I needed 3 months to understand that :) ).
I did start with plain "Scrapy" and it did work fine (maybe some parameters could be explained a bit better especially the ones involving concurrency) and it has been in general very helpful to initially understand some concepts related to web scraping.
But now, after having rewritten multiple parts like 50 times to be able to handle most of what can happen, having split the workload into a multi-staged pipeline involving different specialized programs, having implemented custom code to distribute domains to be scraped to dedicated subprocesses (to greatly improve performance), and much more... I am using Scrapy basically to only download the raw bytes and to take care of what's mentioned in "robots.txt" => my next step will most probably be to replace it (now I don't like a lot having it as a dependency for my program, I therefore started thinking that such a complicated lib is a bit a "risk") with "requests" and something that can help to interpret "robots.txt".
In my case ("wide" Internet scan involving so far ~8M webpages distributed over ~500K domains) I think that it might have been better to start straight with just using "requests".
You can just watch the network tab of dev tools to know what endpoints to hit. Same applies to mobile apps – before you reach for BeautifulSoup, maybe check to see if the website has a mobile app too. It’s usually worth setting up a mitmproxy to see what API you might be able to scrape.
Quite annoying though
But as a general rule, we crawl all we could/know which is a news website.
Can add any niche/industry-specific websites on request.
Can't even imagine that number... different languages or something?
Feel free to contact me. artem [at] newscatcherapi.com
A great example above all is Cloudflare. You simply can't convince Cloudflare you're a human with Scrapy alone. Scrapy has only experimental support of HTTP2 and does not support proxies over HTTP2 (https://github.com/scrapy/scrapy/issues/5213). Yet, all browsers use HTTP2 now, which means all normal users use HTTP2... You get the point.
What we use now is Got Scraping (https://github.com/apify/got-scraping). It's a special purpose extension of Got (HTTP client with 18 mil weekly downloads) that masks its HTTP communication as if it was coming from a real browser. Of course, this will not get you as far as Puppeteer or Playwright (headless browsers), but it improved our scraping tremendously. If you need a full crawling library, see the Apify SDK (https://sdk.apify.com) which uses Got Scraping under the hood.
There's a lot of good things like not having to worry about storage backends, request throttling (random seconds between requests), the ability to run parallel parsers easily. There is also a lot of open source middleware to help with things like retrying requests with proxies and rotating user agents.
However, like any battery included framework it has downsides in terms of flexibility.
In most cases requests and lxml should be enough to crawl the web.
Don't mind my `curl | pup xmlstarlet grep(!!)`s... Nothing to see here...
However, the jobs were first come, first serve so he was waking up at 4 am and constantly refreshing for hours trying to be the first one.
When I heard about it, I quickly whipped up a `curl | grep && send_notif` (used pushback.io for notifs) and it helped him not have to worry so much.
When a new job posting finally came along he was the first in line and got the job :)
I ran it every minute.
I had to include a unique uuid to cache bust, but other than that there wasn't any state.
I wrote about it here https://plainice.com/webscrape-notifications
I guess it’s a matter of preference
The easiest scraper with a proxy rotator I've found is in my current fave web-automator, scraper scripter and scheduler: Rtila [1]
Created by an indy/solo developer-on-fire cranking out user-requested features quite quickly... check the releases page [2]
I have used (or at least trialled) the vast majority of scraper-tech and written hundreds of scrapers since my first VB5 controlling IE then dumping to SQLserver in the 90's and then moving to various php and python libs/frameworks and a handful of windows apps like ubot and imacros (both of which were useful to me at some point but I never use those nowadays)
A recent release of Rtila allows creating standalone bots you can run using it's built-in local Node.js server (which also has it's own locally hosted server API you can program anything else against using any language you like)
"‘Cause if the web server said no, then the answer obviously is no. The thing is that it’s not gonna say no—it’d never say no, because of the innovation."
While there are scraping APIs that unblock requests and charge for them, I'd love to learn more about how they work....
One of the most underrated features is the request caching. It really helps with the problem of finding out your spider crashed or you didn’t parse all the data you wanted and rerunning the job. Rather than making hundred or thousands of requests you can get them from the cache.
One nitpick is that the documentation could be a bit better about integrating scrapy with other Python projects / code rather than running it directly from the command line.
Also, some of their internal names are a bit vague. There’s a Spider and a Crawler. What’s the difference? To most people these would be the same thing. This makes reading the source code a little tricky.
1. Mixing the terms "scrape" and "crawl". Crawling means following URLs found in a page, automated browsing. Scraping means extracting data from a page. Pages can be "scraped" without doing any crawling. The difference between writing a "scraper" and a "spider" is significant. The later is far more complex.
2. The OP gives the reader a peek at what some Python code looks like but it does no show us what the extracted data and its preferred output format looks like. If someone reading wanted to quickly compare this solution to some other solution, e.g., "can our solution do what this one does", they would need to see what the output data and its format looks like. To test we need to know (a) the input, i.e., the example webpage, and (b) the output, i.e., the extracted data including any desired formatting. Here we are provided with (a) only.
1. Instead of creating a urls global variable, use start_requests function.
2. Don't use beautifulsoup to parse, use CSS or XPATH.
3. If you are going into multiple pages over and over again, use CrawlSpider with Rule.
It's way faster and has better support for CSS selectors.
This is my current setup as well, been scraping on and off for 20+ years now.
By "not Colly" I just wanted to remark that in Go is relatively easy to write a crawler from scratch.
That being said there isn't anything wrong with using Scrapy for this. If you're more familiar with web browsers than Python something like https://github.com/puppeteer/puppeteer can also be turned into a quick way to scrape a site by giving you a headless browser controlled by whatever you script in nodejs.
After scraping tens of millions of pages, possibly hundreds of millions, i've fallen back to LXML w/ Python. It's not for all use-cases, but it works for me.
One thing I'll attempt to do before scraping the page is look to see if the page is rendered server side or client side. If it's client side, I'll see if I can just get the raw data and if that's the case, it makes it much much easier.