If you yelled at a deaf person to go away, are you surprised that they don't comply?
If I am facing them, and they can read lips, then yes.
robots.txt has no legal standing by itself, but is a convenient way to translate "terms of service" (which do have legal meanings, but they're often more limited than people assume) into something that bots can automatically understand. You're under no legal obligation to follow robots.txt, but it's a tool to prevent stepping on toes.
The "robots.txt" file has figured in two copyright cases.[1] Courts have taken notice of it as an indication of the web site operators's intent.