The Rise of the AI Crawler

The inefficiencies in the crawling for these AI products is surely going to get better in a hurry, they'll be burning through resources/money as it stands.