From the github issue,
"it would be a fair bit of extra work for us to read robots.txt", so clearly tracking commit hashes would be even more work.
"it would be a fair bit of extra work for us to read robots.txt", so clearly tracking commit hashes would be even more work.
No comments yet.