Include a referrer HTTP header, set the user agent, and set a random wait time between 500ms and 5s.
I crawl Google frequently and that format has always worked for me... At least it did until I made this post. :)
I crawl Google frequently and that format has always worked for me... At least it did until I made this post. :)
Google has a lot of conflicts with media companies and it always starts out with google ignoring some "rules" expecting to either win in court or settle at some point.
So I think breaking these rules is part of the process of how sensible rules are established in the first place. Yes it's recursive, but HN readers should be smart enough to understand that ;-)
Why not? Google obeys robots.txt -- if you don't want them to crawl your site, it is trivial to arrange. I think violating Google's terms of use is pretty hard to justify as ethical.