That's not the server's problem. As the server, they should not be inconvenienced for a problem caused by someone else. The users (which includes the bots) should pay the price.
What percentage of websites that you use is that, exactly?
That's fair. I think your solution would work for those.
https://continuations.com/post/180985743645/world-after-capi...