Hacker News Reader

Comment by gjreda | Hacker News Reader

Do you guys have any concerns about websites' terms of use restrictions?

I've done a bit of crawling and have always been curious how selling this sort of service would handle those potential use restrictions / legal issues.

View on HN

I've been building a product search tool. I wanted to scrap data to power it. It'd be similar to the caching and scrapping Google does.

But I spoke to a lawyer and she strongly advised against it (she said if the Terms of Service of a site say it isn't allowed, you have to follow that).

Dunno. Confuses me how something is supposedly illegal yet many companies do all the time without a concern.

These people seem to be making a decent business out of selling scraped data - http://www.aggdata.com/ There might be others too.

Yes, wouldn't you be a legal target for many companies once they have discovered that you are scraping their information? I've done crawling myself and wonder if I can actually resell the data without being afoul of the law.

The answer is complicated, because the law isn't shaken out yet. Doubt it will be for centuries.

I've written "Is scraping legal?" here, with details, and my own view morally. https://blog.scraperwiki.com/2012/04/is-scraping-legal/

Don't put information on the public internet if you don't want the public to read it. You can keep your private information nice and safe behind a firewall, or on a private network not attached to the internet. Easy peasy.

Reading is very different from republishing.

I believe you have to look at robots.txt file. If it lets you crawl, you should be fine.

Does the robots.txt have any legal significance? I would think that the Terms of Use can be more thorough and are more legally relevant.

Reply on news.ycombinator.com