I am confused where this traffic is coming from. OP says it's from well funded AI companies. But there are not such a large number of those? Why would they need to scrape the same pages over and over?
Or is the scraping happening in real time due to the web search features in AI apps? (Cheaper to load the same page again than to cache it?)