The AI paranoia is getting out of hand. Worrying about bots spamming you is one thing, but discriminating on crawlers specifically because they're from AI companies - and conveniently omitting the difference between a bot that's crawling (and should obey robots.txt) vs. a bot that's acting as user agent (and should not care about robots.txt) - isn't just poisoning communication; it's setting the commons on fire.
See also: The Dog in the Manger.
My issue isn't with blocking bad-behaving bots - it's with singling out LLMs (both training and use), or worse, assuming the problem is being associated with AI and not bad bot behavior.
Before this LLM craze, the biggest crawlers were search engines. They had a motivation to not bring down their targets, because who needs an index full of dead links. With LLM crawlers, all you need is text, and if the site is forced to shut down because of you, that's just less data for your competitors.
Rather than the AI companies turning up to the common pasture and starting to strip mine as fast as they can despite the protests of other commoners who were sustainably grazing their animals on it?