"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare."
I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.