A guide to preventing Webscraping

github.com

34 points·soheilpro··5 comments

If your data is reasonably valuable, it will be impossible to deter a minimally invested developer armed with puppeteer (with some modifications), residential proxy services, and captcha solving (way less than a cent per captcha). Most sites that attempt to do it hinder their users more than they do the scrapers.

Agreed 100%.

Basically anything that would stop a scraper from downloading or interpreting a single page will also get in the way of search engines and screen readers. The former is generally not desirable and the latter may actually be illegal for some organizations. Just give up if this is your goal.

Rate limiting and other approaches to restrict bulk downloading of your entire dataset are more practical, but are still generally easy to work around if one is sufficiently determined unless you require authentication for all access.

>>> Check the Referer header

Warning on this. Many browsers always leave the referer empty nowadays. A user with no referer is very likely to be a human.

that guide teaches to become too aggressive which is unnecessary for many case. But in turn you are damaging user ergonomics. If google start showing captcha on every search yes they protected it from bot but is that what normal real user want? No.

If you data is that much valuable offer them at price so bot shouldn't be issue in many case?

But these days random eCommerce website asking for captcha is totally ridiculous. Some alibaba website are so frustrating that i don't even browse them anymore.

one solution omitted: provide an API so scraping is unnecessary

Reply on news.ycombinator.com