The real question is that why haven't we moved robots.txt to DNS TXT records? This way the consent to scrape would be crystal clear even before they connect to the server.