Hacker News Reader

>if a site robots.txt file prohibits bots from gathering data,

Pages in a robots.txt file still get put into Google's index they just don't get parsed. Surprised me to find that out; http://www.seomoz.org/learn-seo/robotstxt (see "Why Meta Robots is better than robots.txt").

I think it's an interesting question, but I think it will be technically difficult to differentiate between a person using data (say, with a plugin that indexes everything they browse, or a separate program that indexes their browser cache) and something like the Bing toolbar.

Good question. robots.txt was designed for an age before ajax and the proliferation of plugins/toolbars/etc. It's a valid shortcoming you've identified, imho.

Agreed. That ambiguity is why Google's PR response makes sense.