Comment by andrenotgiant | Hacker News Reader

Comment by andrenotgiant | Hacker News Reader

Parent Full thread

The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google

In my experience, these two use detection algorithms that could best be described as "pattern-based."

Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or forced to fill out a CAPTCHA.

For example, you can hit Google Search once an hour via a painfully obvious scraper no problem. But, even if you take control of a Chrome browser via selenium, and write your scraper to do everything exactly like humans down to typing in the searches and moving the mouse around and clicking results, Google is ridiculously good at identifying bot vs human traffic patterns.

I think the algorithm builds a "normal usage profile" for a combination of IP,cookie/user,device and sets a threshold, activity above that threshold gets flagged.

I can trigger Google's Captcha simply by using Chrome's Incognito Mode plus a SSH-based SOCKS proxy. No automation at all. Which is probably against the "normal usage profile" build for data Google gets from me despite Incognito Mode. Well, for starters, dropping Cookies all the time is probably an indicator. Not sure if they can somehow see my SOCKS tunnel - I hope they can't, but am not basing my security on that.

I'm assuming that your SOCKS tunnel endpoint is on a VPS, or some hosted solution, if that's the case, this probably explains it since those IPs are not the source of typical "user" web traffic. I get the same when I use a VPN.

This type of protection is also aligned with what I think of as the most legitimate concern, which is resource exhaustion. I don't understand why some site operators object to scrapers even when they are well-behaved, and collecting public information. I think if the load is low, let scrapers scrape.

My guess is that the pattern is simply "too many requests from a single IP" plus a manually maintained white list.

When working from within the network of a large company that uses a proxy I saw the CAPTCHA regularly for some time.

Of course the CAPTCHA gets answered mostly correctly in this case, which could trigger manual inspection and finally addition to the white list.

I also think it doesn't have to be that complex.

I once scraped Google Search a few thousand times, in a few seconds, a few times, just for fun, and in a school (1000+ people). It was quite funny when everyone suddenly saw a captcha.

Then a month ago or so I scraped Google+ for some statistics, maybe 1-3k times in 2-3 minutes and nothing happened.

The point is that my node.js scraper was far from perfect. Just a fake Chrome user agent, nothing else.

If a large organization with a static IP has a lot of users, Google is clever enough to figure out that it is large and has a high average number of requests by e.g. seeing that many people use their distinct Google Accounts.

I think that is overestimating their efforts. To build such profile and to monitor each and every activity would be very costly.

I think what happens is that when they detect too many requests from an ip address in a short interval amount of time they throw a captcha.

There has to be some calibration or customization beyond IP. Traffic from University IP vs my home looks very different.

Reply on news.ycombinator.com