I'm absolutely pro AI-crawlers. The internet is so polluted with garbage, compliments of marketing. My AI agent should find and give me concise and precise answers.
All this attempting to block AI scrapers would not be an issue if they respected rate-times, knew how to back of when a server starts responding to slowly, or caching frequently visited sites. Instead some of these companies will do everything, including using residential ISPs, to ensure that they can just piledrive the website of some poor dude that's just really into lawnmowers, or the git repo of some open source developer who just want to share their work.
Very few are actually against AI-crawlers, if they showed just the tiniest amount of respect, but they don't. I think Drew Devault said it best: "Please stop externalizing your costs directly into my face"
Sites will have to either shutdown or move behind a protection racket run by one of the evil megacorps. And TBH, shutting down is the better option.
With clickthru traffic dead, whats even the point of putting anything online? To feed AIs so that someone else can profit at my (very literal) expense? No thanks. The knowledge dies with me.
The internet dark age is here. Everyone, retreat to your fiefdom.