Hacker News Reader

The world's largest scraper is complaining about being scraped...when Yelp and all the rest of the local businesses directories realized they were being scraped and the data was being used to google's advantage, it doesn't appear that Google gave two shits. Tables have turned, and it's like someone stole the handball from the recess yard.

View on HN

> when Yelp and all the rest of the local businesses directories realized they were being scraped and the data was being used to google's advantage,

http://yelp.com/robots.txt

Not a mention of GoogleBot.

well, they're presented with either 1) killing their organic search traffic by blocking Google or 2) allowing the crawlers but not being able to stop their data from being aggregated and included in local results.

Yelp's API specifically says their data cannot be aggregated with other sources. When presented with that, Google simply said they weren't using the API, they were scraping.

Apparently scraping is a free pass to do whatever the hell you want if you're Google.

Reply on news.ycombinator.com