"disobeys robots.txt" is more of a feature