geoscripting

For a quick, one-time job, it works very well, and you don't have to bother configuring SSH/FTP for it :). Of course, one could build a lot on top of DRb, I just wanted to show how easy it would be :)

That will come as soon as I learn more about Fantom

strict is a standard perl module. HTML::TreeBuilder seems to work just as well with malformed HTML.

I used Selenium or WebDriver for that. Not bad.

Most of the coders are used to work solely with an IDE, without ever knowing how things work in the background. I know this is something basic for people that know how an IDE works, but most of them don't.

You could give HtmlUnit a try.

It is recommended that you use the API that google provides for searching, but I fail to see why telnetting would be illegal. After all, both firefox and telnet use sockets to do their job.

Yes, I have worked with mechanize before. I was using mechanize even when there only was the perl version. I added a comment to the article explaining my choice.

The example wasn't written in Java for performance gain. It so happened that I had NetBeans open , and it was easier for me to write it in Java at the moment :).

There's also mechanize for python. It can be found here : http://wwwsearch.sourceforge.net/mechanize/ . It handles cookies by default, and is a pretty good tool.

What did you thought this will be about? :)

More and more websites are adding Turing tests after relatively short intervals of time. I consider that to be a good thing, and there are some good ways to stop/identify spiders. You can check out this site: http://stackoverflow.com/questions/450835/how-do-you-stop-sc... . It shows some pretty good techniques that one can use to defend from spiders.

Twill is an option indeed, but this way you don't miss out on the javascript. You can take advantage of all of your browser's features.

I think you misread the article.

That's how it all starts :)

True, you can use wget as well, but this way you have more control over your code.