Comment by bambax | Hacker News Reader

Comment by bambax | Hacker News Reader

> But Yahoo's security makes scraping a challenge. Every time I've tried to back up pages, I can only grab a few files with curl or httrack before Yahoo starts serving blank responses. (...) If you have any idea how to scrape Upcoming's events

A way to bypass security is to distribute scraping over many, many clients.

For example, one could write a browser extension and have many people use it at once; this would have the added benefit of being able to scrape information generated by JavaScript (just do the scraping after the page is completely rendered).

If needed, I can help write such an extension for Chrome (I had written one for del.icio.us some time ago that did exactly that, and another one for IMDb).

(There would be a public server that would serve the ids of events yet to scrape, and that would receive the data that has been scraped).

But for this strategy to work, you need a big community of people willing to help (people who will download the extension and let it run).

Also, 11 days is a very short time...

ArchiveTeam already has a system for this. It's called "the warrior" and it's a little VirtualBox image that checks in with a centralized tracker to get jobs and upload rescued data. You can see the progress on the "upcoming" project here http://tracker.archiveteam.org/upcoming/

Ok, I'm downloading this as I speak.

Still, it's much more complex than a simple webapp (so the audience willing to do it will be smaller). If there are ~12 million events, we need 10 000 people downloading 1200 pages...

(On the leaderboard, the number of "to do" items goes up instead of down: what does it mean?)

I don't know for sure, but I expect it's a standard thing from any spidering. Let's say you start at one page and find 10 links; your tracker will then have 10 items. If each of those pages has 10 links, then you can then have up to 100 items to download. And so on.

Over time you're going to start seeing duplicates (ie. things you've already archived/spidered), so the number will go down. But while you're still in the 'discovery' stage it's going to go up and up.

Oh, ok, if you're spidering.

But here's what the OP said: "All of Upcoming's events and venues use autoincremented ids, making it dead simple to generate a list of URLs to scrape."

So if you go from 1 to {current_id} you get them all, and you have the list at the start.

Sometimes the "items" can get huge. Usually one user counts as one item, so some users run into multiple gigabytes and take days to finish. How would you handle that with a webapp?

I'm not sure about the details for this project, but sometimes it's not easy to get a dump of a list of the items you want all upfront. So the warrior process catalogs new items it encounters and updates the list on the tracker.

Ok, I get that now; but in this particular project every event id is known (see my other comment below).

A webapp would simply need to receive a list of ids and download one at a time (with a comfortable delay between each call so as to not trigger Yahoo's security).

Reply on news.ycombinator.com