>They will limit your local crawler from accessing pages. They will demand you use their crawler.
Source? According to cloudflare, their crawling service don't get any special treatment from their WAF/CDN.
Source? According to cloudflare, their crawling service don't get any special treatment from their WAF/CDN.
And what, I'm here to do research for your arguments?
Cloudflare provides services that any of their users may enable to charge crawlers for access. And they announced a while ago that this was going to be enabled by default on new domains, whether the customer asked for it or not (and there is plenty online about people wondering why they aren't being crawled). When it is enabled, Cloudflare either blocks, require registration of the crawler (forcing enabled billing), or provide one themselves: https://developers.cloudflare.com/changelog/post/2026-03-10-...
My compressed claim was close enough (in my opinion), given the substantive argument was broader: chokepoint, MITM, decryption, the IP vs AI tension.
It's the same pattern as Flock cameras quietly becoming the default license plate surveillance layer across US cities. Private infrastructure is becoming the substrate for something that used to be public or distributed, with the concentration happening faster than any deliberative process about whether we want it.
I'm willing to be wrong on any specific piece of this. But chasing down receipts for every line is effort, and I'm tired and old. The shape is what matters, and the shape is a man-in-the-middle.