I understand that the robots.txt is mostly taken as guidelines for courtesy. But if a large corporation can not be expected to respect it, there might be more proactive measures applied as countermeasures.