Everyone forgets that websites become almost unusable because of crawlers and bots.
Website owners specifically choose for cloudflare to protect against this, it's not forced upon them by cloudflare.
Website owners specifically choose for cloudflare to protect against this, it's not forced upon them by cloudflare.
Could anybody still create a new search engine nowadays?
https://radar.cloudflare.com/traffic/verified-bots
https://blog.cloudflare.com/friendly-bots/
> At Cloudflare, we manually “verify” good bots, so they don’t get blocked.
But cloudflare references robots.txt a lot ( which i mentioned before => to respect robots.txt)
Additionally, they solve the authentication problem here. As a website owner that got bad crawlers ( that copied user agents), i just whitelisted Google's IP's and blocked all the other crawlers.
It seems that cloudflare is actually fixing this problem and making competition for Google possible here.
I recall an article that Bing actually circumvented robots.txt a bit, because site owners were only allowing Google and blocked all the rest => Bing. Which gave Google an unfair advantage ( searched for it, couldn't find it)
Similar article to highlight the issue: https://www.fastcompany.com/90709672/the-little-known-reason...
Our opinions about this seem to differ severely. Cloudflare actually enables good bots to start competing ( while respecting robots.txt).
Eg. If you live in a dictatorship and use a VPN. You're traffic is together with a lot of people.
The website owner can disable cloudflare their checks and that will leave their site unprotected. The choice of that is up to the website owner, no?
It really sucks for a third party to claim you're a bot and as a result you lose access to resources. And the potential for harm is only increasing.
Worse still, the false positive rate appears unacceptably high (look at all the people here with substantial issues) and there's no recourse unless you get a highly voted thread on hacker news.
And their technical skills probably skews perception ( exotic browsers, behavior, VPN, Tor, ... )
As such, can you say, by indication, what percentage of web surfers "a lot" is? ( = Including non technical)
I don't disagree that people can have problems. I'm just wondering how representative "a lot" in reality is.
Yeah but what sort of transparency does Cloudflare offer website owners about what kind of traffic was blocked and *why*?
I’ve had to deal with these things easily bringing sites down.
If all we had to deal with were the users, wouldn't need nearly the spend on the site. Started manually blocking some of the worst offenders.
Feel free to prove me wrong and disrupt cloudflare by only handling that use-case
https://news.ycombinator.com/item?id=34101988
> Had a lot of spammers with Russian language. Implemented expanding xml-bombs, Google Captcha, hidden input fields and a couple of other things against bots. But the block on the russian language was most effective ( and since I was dogfooding it, I didn't see the harm at the time. But it's out of scope at this very moment, yes).
No one "forgets" this because it isn't true.
There are all kinds of tools that you can easily deal with bots and the large DDOS your ISP can handle for you if you are willing to pay for it.
That's exactly what people are paying Cloudflare for, because contrary to your local ISP they are actually competent at blocking a DDOS attack.
People use services like Cloudflare exactly because they don't want to spend a fortune on complex infrastructure just to deal with abuse. Even a mostly-static page running on a reasonably-specced server can easily be overwhelmed by an attack. Why spend $1000 / month on hardware when you can spend $100 / month on Cloudflare's protection?
Cloudflare actually provides this service for free (for simple use cases at least).
I don't know how to come down on this issue. On one hand, I am against the centralization of cloudflare and the risks that come with it.
On the other hand, cloudflare allows almost anyone to set up a simple website and serve it to large numbers of people with very little resources/cost and advanced protection from DDOS attacks.
Note: light use-cases ( or should I say shared models?). Not heavy GPU tasks
input: This is fucking badass!
result: [ { "label": "NEGATIVE", "score": 0.9994425177574158 } ]