Not that I'm arguing for or against preventing access from AI crawlers, but wouldn't it make more sense to block them at a higher level, e.g. the webserver, and not even give them the choice to obey/disobey robots.txt?
- Cloudflare
- Webserver-level user-agent blocking (Apache, nginx)
- Application-level user-agent blocking (`if request.user_agent == 'OpenAI'`)
None of them are ideal since you can simply change your user agent, but all of them seem like better options than robots.txt to me.
e: Okay, this is funny.
E.g. nginx $http_user_agent