Well behaved crawlers don't.
Products like Google Search and Bing have a self-interest in supporting robots.txt because the alternative is annoyed webmasters suing them for not providing an opt-out due to which their unsecured credentials are a quick search away.
In fact, I believe the robots.txt protocol was devised in consultation with search engines.