ParentFull threadshaunpud·If a site provides a sitemapindex in the robots.txt, would you assume they're allowing their site to be crawled on the pages they've set out within?View on HN