Hacker News Reader

"Using a simple scraper written in Ruby, I was able to grab data on the >90,000 TV shows in the IMDB database"

I know it may be a bit more hard to use than a crawler (sadly), but IMDB give their dataset for free (under conditions) : http://www.imdb.com/interfaces

(imdb stricly forbit crawler)

edit : nice analysis ! thanks !

View on HN

True, but I think the rational behind that is that imdb doesn't want someone creating a software that gets used by a lot of people which implements such a crawler.

I doubt although I can't say for certain, that imdb much cares about someone who made a scraper to gather data for a one time aggregation like this.

Of course, for a use case like OP has, the static data imdb offers would also have sufficed, I'd say. Of course, that data is not as "up to date" as the data you can scrape off of the web pages, but it would have been sufficiently "up to date" for such an analysis, I think.

Reply on news.ycombinator.com