robots.txt is a "sign" that you leave at the door for well-behaving robots, but it doesn't actually make any practical difference when a robot isn't implemented to honor it