ParentFull threadDiti·If one doesn’t want LLMs to scrape data and knows the LLMs will be ignoring the robots.txt file.View on HN