Would someone let me know if I’m just plain wrong in this assumption? I’ve run many types of sites and scrapers have never been anywhere close to the main source of traffic or even particularly noticeable compared to regular users.
Even considering a very commonly scraped site like LinkedIn or Craigslist - for any site of any magnitude like this public pages are going to be cached so additional scrapers are going to have negligible impact. And a rate limit is probably one line of config.
I’m not saying you are necessarily wrong, but I can’t imagine a scenario that you’re describing and would love to hear of one.