it says it'll respect robots.txt if you don't want your page crawled (parsed? interpreted?)