Feel free to characterize your recommendations by whether the intended audience is an experienced dev or a n00b.
UPDATE: If any patterns emerge for pain points or missing docs, examples, or resources for learning I might create a page or an ebook project on GitHub that covers that area.
If you are interested in web crawling, which is often necessary if you want to extract data from very large sites (or many sites), I just wrote up a blog post comparing open source web crawling systems (including scrapy): http://blog.blikk.co/comparison-of-open-source-web-crawlers
In the end, it's the infrastructure that powers the extraction that requires all my attention. I've got a bunch of techniques I use, but I'd love to compare with how other people do it.
Love that more scraping resources are coming online. I see scraping as the important link between the web as it is now, and the web as it will be in 20 years. (web2 -> web3 for the jargon geeks.) The whole semantic web isn't going to be useful for most non-academics without considerable structuring effort put into existing data.
Edit. Thinking about what's available now, maybe something using phantomjs or similar?
Of course this only covers the actual gathering of the content, not storing/indexing in an efficient way.
http://blog.zillabyte.com/2014/06/23/5-easy-steps-to-build-a...
There is also Import.io for web extraction. Do you have something you are looking for in particular?
UPDATE: If any patterns emerge for pain points or missing docs, examples, or resources for learning I might create a page or an ebook project on GitHub that covers that area.
And shameless plug - for anything more trivial and on-demand or websites that have good structure (which is the most of them) I use http://redpluck.com. It's easy and fast to setup, supports scraping behind login walls and powerful deep scraping.
One thing CommonCrawl really needs is a good index. https://github.com/trivio/common_crawl_index is great, but it only exists for the data from 2012. Apparently there was something in the works for 2014 data but there hasn't been any update for months: https://groups.google.com/forum/#!topic/common-crawl/mrZBnvD...
I probably will play around with the tools that are new to me regardless.
UPDATE: Updated the OP.
This general advice should serve:
1. Get the page content the simplest way possible.
2. Use regular expressions to extract the desired content.
For both the above goals, and because so much revision is needed over time, I recommend Python.
Use regular expressions to extract the desired content.
It can search, extract fields, and page through results--automatically (no config). A full-featured developer API will be available soon for finer-grained control.