I have to imagine that's not available to avoid training by commercial entities. https://www.transformativeworks.org/ai-and-data-scraping-on-...
Confusing. Would even prefer a commented crawl delay that specifies in human language since then I could special-case it.