Currently it doesn't look at the robots.txt file. I had taken note of this, and will be added in future releases. Thanks for the suggestion.