Jake232

Most significantly: "Your data isn't even in the picture. We are simply not interested in any of it." - Pretty sure Facebook are interested in it.

From the Github page:

Third, Goji is not magic. One of my favorite existing frameworks is Martini, but I rejected it in favor of building Goji because I thought it was too magical. Goji's web package does not use reflection at all, which is not in itself a sign of API quality, but to me at least seems to suggest it.

Or if you're a python enthusiast, then shameless self link: http://jakeaustwick.me/python-web-scraping-resource/

How so? I'm from the UK and most of the people I know would say MS are a reputable brand for sure.

Although this may seem like a massive mistake, that's because we all associate Microsoft as a negative brand, I think you're forgetting that it's associated with negativity in tech circles only. 95%+ of consumers see Microsoft as a huge corporation they can trust. If you asked your parent whether they think Microsoft is a good brand, the answer would likely be "Yes".

The submitted link works for me.

Somebody asked this on the blog post, here was the reply from Linode:

If you take the upgrade, you inherit the new plan specs, vcpus and all.

We’ve greatly reduced the contention on these new machines compared to our old structure, and in testing this new arrangement provides much more consistent CPU time with less potential for steal. We think it’s great and totally worth the move, otherwise we wouldn’t have done it. These machines are incredibly fast, faster procs, SSDs, the network is incredible, etc.

The increased CPU on some of the plans compared to DO is what is encouraging me to change. When doing web scraping, it's easy to max the CPU once you are doing 20 paths on a page.

I write my own custom scrapers, I prefer the flexibility and feel safer that the service isn't going to disappear any minute.

If anybody is interested, I wrote a detailed article on scraping not so long back that was well received here: http://jakeaustwick.me/python-web-scraping-resource/

I've always wondered why github never had tags.

I see no harm in doing so. I have a ton of books in digital formats, just never thought about buying a physical copy before. The only reason I did so this time is because I have some travelling upcoming, so it's ideal to read on the plane.

First ever physical copy of a programming book I've bought, lets's hope it's worthwhile.

Sidekiq wasn't around when they made resque.

If you're running on a server with 1Gbps, then yes - it can be an issue. Another issue (I'm going to add a section on it) is that you can peg the CPU at 100% very easily. Parsing HTML / Running xPaths uses a lot of CPU, so if you're running a good number of threads this can quickly become a problem.

Hi.

I mentioned selenium in the section below that, but I'll drop a note to it in the AJAX section too!

Thanks for the mechanize link, I'll add a note about it.

Mechanize is awfully slow though, if you need to crawl quickly it's not asynchronous. I wouldn't want to use it for general crawling. I guess you could patch the stdlib with gevent, and try and get something working that way.

Thanks, good to know regarding the proxy. There was a couple of other little things that just didn't work the way I wanted though (I honestly don't remember them now).

I've built private libraries on top of requests now that allow me to do everything in such a trivial amount of time, so I prefer this approach with more control.

I think if I was going to write a long running spider, I'd probably look into scrapy again beforehand.

Hey, thanks for the feedback!

I'm planning on adding more to the article in the near future, this was just a start. I plan on it being a resource with almost everything in, so people can bookmark it for future use.

I really haven't seen lxml choking on pages, are you sure you have the latest libxml2 installed? lxml always seems to work for me. If you know of a URL where it doesn't, I'd love to see it.

Hey,

I've actually built something similar to this myself, I plan on writing an article in the future with something along these lines.

Yours look pretty polished though, good job!

I've wrote a lot of python crawlers, and have never really experienced an issue with lxml. It doesn't really ever seem to choke, and I'm sure I must have come across some pretty funky markup in my time. lxml has always handled things fine for me. Maybe I've just gotten lucky I guess?

Thanks! I mentioned BeautifulSoup existed very briefly in the lxml section in prerequisites. I just added a short section on CSS Selectors, you can actually use lxml with them. I'll add a little more info with BeautifulSoup and PyQuery mentioned though.

I've used Scrapy previously, and agree it's a good tool in the right situation.

It could just be me, but I prefer more control, and using the requests library gave me that. I ran into some obscure problems when wanting to use / change multiple proxies with Scrapy.

I think for a long running spider, then maybe scrapy is worth looking into. For simple one-off scripts / things not run very often, I'd much prefer to write something custom and not have to learn about Pipelines, Filters, etc.

Going to add a section on PyQuery, thanks!

Never used it, but looks useful. I'll add a note regarding it. Thanks.

Hey, article author here.

I've done extensive scraping in both Python and Ruby, as I wrote most of the scraping / crawling code at http://serpiq.com, so I can chip in.

Overall, I prefer Python. That is pretty much solely down to the requests library though, it makes everything so simple and quick. I haven't covered it in the article yet, but you can extend the Response() class easily, so you can for example add methods like images(), links(nofollow=True), etc. Overall, I just think the requests library is much more polished than anything available in Ruby.

grequests (Python) means I can make things concurrent in a matter of minutes. However in Ruby the only capable library supporting concurrent HTTP requests that I liked was Typhoeus. It just wasn't to the same standard though, and I ran across certain issues when using proxies etc.

As far as the HTML parsing goes, I don't really have any preference. Nokogiri and lxml are both equally capable.

I think they're both perfectly capable languages though, stick with what you prefer. I've been experimenting with Go lately.

Any further discussion on the Bloom FIlter anywhere? (an issue, etc).

Second year University of Hull compsci student here, would just like to thank you for this - it's so much more efficient to use than the official intranet version.

Also seeing this.

Google Chrome Version 31.0.1650.57 OS X

I think you should be able to incr/decr by more than 1. Maybe allow http://arbitrarycounter.com/vb/fruit/apples+N where N can be any integer > 0

I feel people have become reliant on GitHub though. They use the web interface to do daily tasks. They like the one click fork to move a repo to their own account.

The problem is, on a daily basis when you're looking for an open-source project, where do you find it? It's always on GitHub. It has established itself as the market leader.

It happens all the time. Other companies may provide better support for certain things(private repos in this instance) - but people flock to where the rest of the market is. It's like PayPal, we all know they're better alternatives and yet we all still use PayPal.