> 1. The bots have essentially unlimited memory and CPU. That's the cheapest part of any scraping setup.
Anyone crawling at scale would try to limit the per-request memory and CPU bounds, no? Surely you'd try to minimize resource contention at least a little bit?