Hacker News Reader

Comment by Mystalic | Hacker News Reader

I wonder if this works every time...

And if it did, how long until a defense is made...

There is nothing magic about web browsers, telent to port 80 at www.google.com and with a simple get request they will spit back their website. You can make it a little harder to do this stuff but a packet sniffer is always going to let you pretend to be any software you want unless they are using encryption. Also because Firefox is open source you can't prevent people from scripting using it anyway.

PS: I recommend all aspiring coders to telent to www.google.com at least once just to feel the magic.

"Firefox is open source you can't prevent people from scripting using it"

There are frameworks for doing that already, check out Selenium: http://seleniumhq.org/

That's how it all starts :)

> PS: I recommend all aspiring coders to telent to www.google.com at least once just to feel the magic.

I'm pretty sure it's illegal in a couple of ways. </bitter>

But yes, the magic is there.

illegal!? why!?

It is recommended that you use the API that google provides for searching, but I fail to see why telnetting would be illegal. After all, both firefox and telnet use sockets to do their job.

Again, the comment above wasn't meant to be taken literally, but just because they both use sockets doesn't mean they're just as legal.

Imagine a web site which terms of service state you cannot use software to circumvent ads. Or where part of the security is done client-side (stupid, yes, but not impossible). Skipping the browser breaches at least the terms of service, and may be constructed as hacking. I think even Google discourages automated searching and prefers you use its api, which (at least some years ago) wasn't free for commercial use. I may be wrong in this particular case, but the important point is you may want to check the specific TOS before skipping the browser.

DMCA, of course. The reply was meant to be exaggerated, I hope telnet itself isn't illegal.

Some sites will expire the cookie after a period of inactivity, in which case this will stop working. But it can easily be refreshed by getting a new set of cookies, which as other people have pointed out is a process that can be automated.

You could make it very onerous to move around the site even if you're logged in. For example, after every few links you follow you have to pass another Turing test. That would be completely unreasonable though.

I think once a user is logged into your site, you'd have an extremely hard time defeating this sort of behavior without degrading the quality of user interaction or treading on legitimate use of your site. You may be able to defeat egregious abuses, such as scraping entire photo galleries in seconds, but even that can be defeated by a script that randomizes requests/times between requests.

More and more websites are adding Turing tests after relatively short intervals of time. I consider that to be a good thing, and there are some good ways to stop/identify spiders. You can check out this site: http://stackoverflow.com/questions/450835/how-do-you-stop-sc... . It shows some pretty good techniques that one can use to defend from spiders.

Reply on news.ycombinator.com