Starting to ignore robots.txt? Unfortunately, there's no way to prove that anyone respects it. A well-identified bot can back off, then return from a different IP address with a different User-Agent and attempt to mimic a human user. Webmasters really have no defense against policy violations. If you run a bot of any kind, including a link checker or SSL tester, please respect robots.txt. If not, be prepared to be identified as malicious and blocked by an IDS.