If it becomes standard to have such a file and it effects their bottom line, could they disregard it?
If the file that were previously honored as consent to use the copyright material is subsequently ignored, wouldn't the content creators take the indexers to court for copyright infringement?