Hacker News Reader

Comment by mbillie1 | Hacker News Reader

> There are so many that just loading, copying, and pasting all of them took the little script I wrote more than 20 hours.

I want to see that script.

View on HN

I laughed at that line. Here is someone who never scraped a website or run a spider to collect data from the net. It's a magical feeling next day, when you see your data.

To people who do their work on computers, but have never programmed before, programming is like a superpower.

It ran on a netbook and was actually controlling a browser and copying and pasting. It's not optimized, but it got the job done, so I'll applaud anyway. Great article, I expect we'll see more like this soon. ;-)

Oh and if you're nice on Netflix's servers, perhaps at 1 request a second, it would still take 25 minutes to run through 90,000 requests without the overhead of the browser. Sounds like they averaged one request every 24 seconds.

It actually looks pretty wasteful, since it probably needed to load the full webpage instead of just the html.

It was enough for the author to write a good story on it.

Wastefulness be damned, it got the job done.

This sort of thing is generally pretty easy. For instance, this would be a good start:

    for i in {0..10000}; do
       wget http://whatever.invalid/genres/$i
       sleep 1
    done

From there processing the results to pull out the genre would be a regex away; a single properly formed grep could probably do it.

Determining the top number would have to be done by hand; having holes in the sequences makes it more complicated but you could still use a sort of basic binary search.

That would be a good start on a site that doesn't require authentication. :)

You can add a cookie jar, probably good enough for these purposes when combined with browser string override.

Reply on news.ycombinator.com