Hacker News Reader

Comment by bad_user | Hacker News Reader

The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice.

You really can't protect against this unless you start making the experience of regular visitors much worse.

View on HN

Interesting, i am facing problem with this approach.There is a site which pops up alert box for username and password and i can't seem to switch to it with chrome web extension.With selenium it is as simple as driver.switchTo().alert(); have you ever faced this problem? by any chance you know how to fix it? I am trying to accomplish the same thing as described in this question in SO https://sqa.stackexchange.com/questions/20710/not-able-to-us...

Sure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc).

For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well.

It's _definitely_ doable, but it's not as trivial as recording a selenium macro. (Not to mention these tools look for selenium presence and extensions anyway).

IF you throttle the connection then wouldn't download time and network latency, etc., give you a better random?

You can generally tell because of the distribution, which tends to show up after lots of data points.

The issue here isn't that it can't be done, the problem being one of false positives.

By doing this you must accept that a certain percentage of legitimitate users, which can be quite significant, will be blocked. In case you're wondering, yes, this does happen with solutions such as Cloudflare.

And at that point, either your website isn't popular, in which case you can't afford to lose users, or it is very popular, in which case you'll piss of enough users as to receive bad reviews.

Basically you can't afford to do this, unless you're Facebook or Google, and you have to then wonder why Facebook or Google do not deploy such protections

So going back to my main point, of course it's possible, but the experience for users gets significantly worse such that you won't want to do it.

I've found Cloudflare's solution to be very inadequate compared to Distil or Incapsula.

Both Distil and Incapsula _did_ have a small number of false positives (showing a captcha to users). We did have to write some code to overcome that.

Yeah, but if you can use ML to pick out the differences between real users and bots then you can also use ML to mimic real user behavior. Both sides spend a lot of time and money trying to overcome each other, and end state is the same it was at the beginning: near total access to the data, at least for those sophisticated enough.

But your crawler must be very slow then.

They generally are fairly slow but it doesn't matter because you just set it and leave it to it until it has all the data you want, leaving you to get on with other things.

Plus very easy to parallelize

Unless the websites work together to protect themselves against crawlers that work that way (e.g., rejecting requests to different hosts from the same origin within a time interval).

Reply on news.ycombinator.com