Maybe there should also be a protocol to support paying a small amount of crypto (on the order of 0.01-0.1 cents per request) to bypass the PoW. The bots can still scrape it by paying, but you now price in the externality.
"proof of pay" or "proof of a transaction"
An advantage of this would be not wasting electricity for the proof of work computation.