Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.
Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.
I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-an...
Plenty of open-source ones as well that could bypass, eg maybe this one that came up in search https://github.com/VeNoMouS/cloudscraper Combine with residential proxies and you're just not going to find them.
> Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.
Just from a quick Google search:
- LinkedIn: https://brightdata.com/products/datasets/linkedin
- Amazon: https://www.junglescout.com/features/product-database/
Just an example: Brightdata linkedin database has 19 million entries. Linkedin has over 1 billion members.
As I said, partial scrapes of small subsets over long time provide no real value for AI scrapers (repeating the main argument).