That makes no sense.
There is no reason for AI scrappers to use tens of thousands of IPs to scrape one site over and over.
That just sounds like a classic DDOS.
There is no reason for AI scrappers to use tens of thousands of IPs to scrape one site over and over.
That just sounds like a classic DDOS.
Having lots of IPs is helpful for scraping, but you don't need 10k. That's a botnet
It can absolutely be that, but that requires a confluence of multiple factors - misconfigured scrapper hitting the site over and over, a big bot net like proxy setup that is way overkilled for scrapping, a setup sophisticated enough to do all that yet simultaneously stupid enough to not cope with a site is mostly text and a couple gigs at most and all that over extended timeframe without anyone realising their scrapper is stuck.
Or alternative explanation: It's a DDOS
Also I don't know why you think this is sophisticated, it's probably 40 lines of Python code max.
This stopping is absolutely not "evidence" that the motive was grabbing data. Honestly...
I have a site which is currently being hit (over 10k requests today) and it looks like scrapers as every URL is different. If it was a DDoS, they would target costly pages like my search not every single URL.
SQLite had the same thing: https://sqlite.org/forum/forumpost/7d3eb059f81ff694 As have a few other open source repositories. It looks like badly written crawlers trying to crawl sites as fast as possible.