robots.txt is merely a gentleman’s courtesy at this point. Nobody is obligated to follow it.