Actually Veritasium has a great video about this. It's proven as the most effective strategy in monte carlo simulation.
EDIT: This one: https://youtu.be/mScpHTIi-kM
Actually Veritasium has a great video about this. It's proven as the most effective strategy in monte carlo simulation.
EDIT: This one: https://youtu.be/mScpHTIi-kM
EDIT: i thought about it for a min, i think in the olden days a crawler crawling every path through a website could yield an inferior search index. So robots.txt gave search engines a hint on what content was valuable to index. The content provider gained because their SEO was better (and cpu util. lower) and the search engine gained because their index was better. So there was an advantage to cooperation then but with crawlers feeding LLMs that isn't the case.
Have you tried Anubis? It was all over the internet a few months ago. I wonder if it actually works well. https://github.com/TecharoHQ/anubis
On their README.md they state:
> This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.
I love the idea!