I don't see us getting a massive innovation in search on the internet now that Google has such a massive foothold, and companies like Cloudflare stop innovation from happening.
How are we "stopping search innovation"?
How are we "stopping search innovation"?
Thanks for taking the time to reply.
You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when Cloudflare suddenly decides it does not want to honour that "agreement"? What happens if/when Cloudflare is sold, or the contact who greenlit these smaller "legit" crawlers moves on and decides that it no longer agrees with said website anymore?
Is a price comparison site a "legit" crawler? What defines a "bot" vs a "crawler" in the eyes of Cloudflare?
Would you need to notify your customers that you now also allow additional crawlers access to their sites, or would they need to opt into it via the Cloudflare dashboard? What happens when you have a falling out with my said company (it happens, relationships sour) and suddenly we can't make contact, then suddenly customers websites aren't being scraped because we're bots?
If you are a price comparison site getting blocked by cloudflare, site owners may be losing sales, and that's good feedback.
Depending on the industry, etc..
I really can't think of a good solution. But that's the tricky position Cloudflare is in - how does it balance everything.
How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl?
Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.
And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
They're only a gatekeeper because sites voluntarily enter into commercial agreements with them. There's no coercion or manipulation like Google AMP.
When I had a site that had millions of pages, I found that sites like Baidu would crawl my site as often, if not more often than Google.
I already felt the relationship with Google was parasitic, but I looked through my logs and never found a single hit that came from Baidu and many of the other search engines that would overload my site.
I was looking at a substantial part of the site running costs going to supporting web crawlers that were not doing anything (1) to help me, or (2) to help end users (if they don't want to send Chinese users to an English-speaking web site, why crawl the site?)
So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition.
There are web crawler behaviors that are annoying: harvesting email addresses, overloading your site, etc. But how do you know who is doing something wrong with the data and who is just collecting it do do nothing with it? (Probably 95% of web crawling ex. Google.)
This sounds like you're onto a reasonable “legit” factor: does the crawler honor robots.txt? Baidu would be legit because they don't lie about their identity and if you put a rule in your robots.txt file they'll honor it.
Yes, I've never understood why it's seemingly so important to CAPTCHA me before serving me less than 100kb of read only plain jane HTML. What sort of "attack" is this stopping? I'm pretty sure the CAPTCHA itself is bigger than half the sites it blocks me from reading.
No such filters should exist. Is it really that awfully bad without anything but basic filters (ban an IP for flooding)? Are there like, operations that try and spam every single Cloudflare-hosted website 24/7?
Legitimately curious if your anti-bot measures come from actual bad experience with the internet or is it just a liability limitation move? (Namely to reduce potential suing surface by angry data owners and/or three-letter agencies.)
---
Basically, if I am experimenting with a basic crawling program and I hit websites A and B 20 times each in a space of one hour, is that really deserving of a captcha or extra auth methods?
Not flaming but I am really curious. Do you have any data and rationale posted somewhere that go into deeper detail about why Cloudflare's bot detection is how it is?
That isn't to say that the customers are savvy and have a good understanding of different types of automated traffic and which automated traffic is harmful and which is benign. Many have a quite naive understanding that doesn't extend beyond "bots = bad, unless it's Google" and dial protection settings to the max for no good reason.
Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
Which makes "contact us for an exception" a no-go, as the relevant source IPs will constantly be changing.
Recently your company announced partnership with Internet Archive, but if CloudFlare want to continue play a role as behevolent party everyone should have equal access to this data. Yeah it means that some bad actors will be able to easily scrap the web too, but...
CloudFlare service can't prevent scrapping anyway. There are shady residential proxy networks, services to bypass captcha and scrapping software like Zennoposter. It's possible to make scrapping more expensive, but bad actors don't care because they have money. Unfortunately enthusiasts, open source projects and small companies don't have enough resources to do the same.
I think you might not understand that it's site owners like me who want to stop scraping. It usually comes from specific bad incidents, like copycat sites stealing our content and work.
Cloudflare wouldn't block scraping if website owners didn't want it. And website owners can easily disable this protection.
Also I'm talking from experience because I been on both sides of the fence: doing scrapping and implementing protection. So yeah your CloudFlare protection will deter 10% of bad actors, but will also cut off 99% of enthusiast / research efforts or users of niche software or browsers. Still anyone with $1000+ budget will scrap whatever they want.
If this doesn't at-least meet your definition of "reach a human at Cloudflare", I'm not sure what will.
That said, I don't think it's a good situation that this is the solution rather than a proper, documented position that people can work to.
In fact it's a textbook definition of a non-scaling solution.
Individual companies having so much power gives me the willies.
But let's suppose someone is building a new cool search engine and our ML stuff is blocking them. Then... contact us/me.
(but Google is allowed because Google was first to market)
Yes, you must adhere to the controls that site administrators put in place, like Cloudflare.... You don't get to blast my site with requests, just because you want to...
(b) But you’re a-ok with Google doing this. Gated communities aren’t really good for anybody but I see what you are saying.
The same is true online. Apple's walled garden has kept hundreds of millions of people safe on their device. It's why iOS malware isn't a thing.
> Cloudflare stops much more than just blasts
Exactly. There's even more benefit to Cloudflare than just DDoS. Captcha's for stopping credential stuffing, for example.
But hey if I pay Cloudflare enough, then I’ll get to blast your site and possibly stuff creds at the same time :/
[1] - https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...
So no problem, time to move on web search is no longer exciting