> what's to stop a spider from indexing it?
a robots.txt that forbids a particular spider from accessing the content.
This assumes that the spider does respect the robots.txt, which googlebot does.
a robots.txt that forbids a particular spider from accessing the content.
This assumes that the spider does respect the robots.txt, which googlebot does.
I don't know. I'd imagine that there isn't a law specifically written for robots.txt and scraping.
If a case went to court I'd imagine it would be settled based on laws meant for things like "keep out" and "No trespassing" signs:)
Currently, it gets used for more than "deny/allow all" - it is selective. If I have to worry about stuff I don't want indexed being indexed, then I'm going to block all crawling. Yes, I know that wouldn't be easy, but I wouldn't be alone in my sentiment.