Hacker News Reader

> Better to ask beforehand, many websites are quite happy to hand you data

If they say "no", what then? If it was within your rights to scrape the data, and you go ahead and do it anyway, then you're going to (a) look like a jerk, and (b) if the company is litigious, they're going to use this as evidence that you knew what you were doing was wrong.

I'm not saying that you shouldn't ask, but pointing out that there are sometimes downsides to asking.

Scraping a website is absurdly easy. You can have the data before you'd have finished writing the email.

But it's not very polite to do (someone runs that website that you're hammering), and it doesn't open you up to a conversation that might get you even more access/data than you originally thought was accessible.

You don't have to hammer a website to scrape it. You can write scraper programs to limit the number of HTTP requests per second.

As long as you obey their robots.txt, who cares? Google is scraping them far more often then your one-off script is going to.