Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea
herrjemand.medium.com
herrjemand.medium.com
I hope that we have something in the future that does this job very well so that services do not need to verify phone numbers, Google accounts, and even IDs and facial imagery just to allow someone to use them (as this is much easier to do than coming up with new captcha styles that humans can quickly and easily solve, but that basic machine learning and scripting cannot).
Being able to use the Internet with the slightest bit of privacy is already ~impossible for the average user and extremely difficult and tedious for very knowledgeable and experienced ones, so anything that tries to improve the current trend sounds like it's at least attacking a problem worthy of our attention.
Instead of trying to solve anonymous human verification we can as well make micro-payment an option.
It seems like responding to a bot and responding to an unpaid accidental request would take the same amount of energy from Cloudflare. The only difference is that a repetitive unpaid request is probably easier to detect.
Intuitively, you would provide privacy-preserving offline attestations that e.g. one of a set of trusted parties has verified that you're a legal resident of jurisdiction X or over Y years of age, without needing to disclose any private identifiers.
I personally simply close the tab when I see a cloudflare "one more step" page.
What does your cdn solution look like?
Route optimization from your (single) endpoint to clients literally half a world away?
> What does your cdn solution look like?
> Route optimization from your (single) endpoint to clients literally half a world away?
And as a developer, I don't understand this newfangled obsession over CDNs either. Yes, there will be 200 ms RTT in some cases. So what? Get over it. Optimize your website to load in fewer round-trips. TCP congestion control adapts well enough to any latency. RTT only really matters in gaming and VoIP.
Cloudflare prevents a bunch of crap that site operators just don't want to deal with. Especially for smaller sites that are run by one person. Dealing with a wordpress site getting hacked because you missed an update by a day, or a bulletin bored getting swarmed with bots, or some asshat ddos'ing your site because you banned them. Suddenly that site just isn't worth running.
Complaining that about a thing that prevents that headache because it's a minor inconvenience to you is so self centered it boggles the mind.
> Dealing with a wordpress site getting hacked because you missed an update by a day
Maybe don't use something this vulnerable then and rely on a third party to protect you from exploits.
> or a bulletin bored getting swarmed with bots
Maybe require email verification and/or a captcha when signing up or posting. Don't punish people for passive actions.
Somehow, there are many forums that aren't behind cloudflare, yet there are no spam bots.
> or some asshat ddos'ing your site because you banned them
Sure ddos is such an everyday occurrence?
I just don't understand. I run a personal website. There's literally nothing to "deal" with. I set it all up once and it works. I only have to pay for the server and for the domains on time.
You're definitely overestimating the technical expertise/available time of a lot small time admins out there.
You don't see bots and spam on those forums either because they are actually using cloudflare, and you're just not seeing the captcha, or because in the backend they're feeding all their posts through akismet (in plain text). I don't think you're considering how many services see your posts, even when you don't trip a captcha.
email accounts are trivial to sign up for, especially for bots. I always recommend charging $1 (or local equivalent) for an account, that's a lot harder to fake.
My point in all this is that bitching that site is using cloudflare to not have to deal with crap, is a self centered view.
Saying "well it never happened to me, so it must never happen" is similarly self absorbed.
Maybe consider that your experience is not everyones experience
Who is serving whom here?
If a business thinks it's ok to impose cloudflare inconvenience on me, the customer, for the priviledge of giving them my money, who is self centered here?
The simple answer is I'll close the tab and go buy it from a competitor. I'm not playing captcha games to buy something.
>I don't see any alternative services with this level of ease of use.
Both of these boil down to laziness, IMHO. And for how awful they are for half the internet, that's not okay to me. Not as a user or as a developer.
When a user requests a site they expect to get the site. When you utterly disregard their utterly reasonable expectation and force them to train someone's ML algorithm for free for a minute or two you utterly destroy the user experience.
If that's my experience as a user, your entire site gets an instant 1/10 rating from me. I'll likely leave instantly and never return.
I've see much worse times for users, and a cdn absolutely help with our staff in asia dealing with our internal apps.
Of course they're not always tripping up cloudflare and being shown captchas. I almost _never_ see a cloudflare captcha either... huh...
Because this implies that.
Page load speeds by themselves can be painful. Open devtools and have your browser throttle to poor 3g speeds.
Try browsing around. even well optimized sites.
Now try uploading a couple dozen files through an api.
This is legit what some users deal with. In New York state even, you don't need to go that far to find poor connectivity.
Even if all your users have awesome home connections, think sales people taveling to a client. or on site inspections of a manufacturer in a warehouse that's mostly metal and has bad wifi.
200ms latency isn't that bad, but I'm seeing more 800-2000ms latencies with some users depending on physical location. at some point latency kills usability. Especially when trying to get through a complicated QA or inventory process.
I thought you were giving an example of how bad connections can get, and saying that the extra latency would make it worse, but in that situation it's a drop in the bucket.
You're missing a zero in that RTT for users in places like Asia if your server is anywhere in the west. (It's actually somewhat revealing when someone throws out a number like this without any qualification; what exactly made you conclude 200ms is the magic number?)
> Optimize your website to load in fewer round-trips. TCP congestion control adapts well enough to any latency. RTT only really matters in gaming and VoIP.
You don't need a very big imagination to think about cases where RTT will have significant impacts e.g. in the event you need to issue multiple sequential requests that are dependent on one another. These are unavoidable and occur often in more than just websites, but anything that e.g. uses HTTP as an API (a very simple one is something like recursively downloading dependencies.)
This comes across as a classic "I don't actually understand the problem domain very well at all, but get off my lawn" answer to the problem.
Well, it does say 130 ms in here: https://www.quora.com/How-long-would-it-take-for-light-to-fl...
And that's around the planet, to go around and end up at the same spot. In practice, with sanely-configured routes, your packets should never need to traverse more than half that distance. So, divide it by 2, then that cancels out because RTT is a measure of how long it takes for a signal to travel back and forth. You then add some time on top of that to account for buffering and processing in the various equipment along the way.
> These are unavoidable and occur often in more than just websites, but anything that e.g. uses HTTP as an API (a very simple one is something like recursively downloading dependencies.)
If you mean REST API requests, the kind that trigger some code to dynamically generate a response, how would a CDN solution like cloudflare help? The request still needs to get to the server and the response still needs to come back, all the way, because that's where that code runs. CDNs only really work for cacheable static content, don't they? I mean it's in the name.
A blog or a news website certainly doesn't need a CDN.
And switching adds significant delay, especially when you get out to the edge.
> If you mean REST API requests, the kind that trigger some code to dynamically generate a response, how would a CDN solution like cloudflare help? The request still needs to get to the server and the response still needs to come back, all the way, because that's where that code runs. CDNs only really work for cacheable static content, don't they? I mean it's in the name.
For a simple example, imagine assets that are dynamically loaded based on feature detection in JS. All of the assets (js included) can be cached on the cdn.
If you have a fiber backed, all-switched network with no routing, buffers, congestions, or detours, you may get that value, if you're lucky.
Pinging tty.sdf.org which is a direct access shell service in USA from somewhere between Europe and Asia, from an academic network backbone roundtrips in ~190ms. I'm traversing a little less than half a globe with the whole journey. In your terms, it should be around ~60ms, but it's not.
> If you mean REST API requests, the kind that trigger some code to dynamically generate a response, how would a CDN solution like cloudflare help?
By using Cloudflare workers, so your code is also distributed around the globe?
> CDNs only really work for cacheable static content, don't they? I mean it's in the name.
JS files are also static content. Unless you don't use code distribution like Cloudflare workers, using a simple CDN can cache 90% of your site if not more. CSS, images, JS, HTML, you name it.
> A blog or a news website certainly doesn't need a CDN.
Actually, CDN is the most basic optimization for distributing heavy assets like videos and images, which news websites use way more than text. Why not use a CDN?
Actually I get around 200 from Russia which is also "somewhere between Europe and Asia":
round-trip min/avg/max/stddev = 196.504/197.581/199.833/1.360 ms
> By using Cloudflare workers, so your code is also distributed around the globe?Great, let's give that company even more control. That's sure gonna end well.
> CSS, images, JS, HTML, you name it.
It all gets loaded once and then cached in the browser. The initial load takes long regardless of whether there's a CDN. Oh, and many websites also use stuff from like 10 different domains, which doesn't help this either.
And, it doesn't matter whether a JS file loads in 50 ms or 300 ms, if it then takes 5 seconds to parse and start running.
> Actually, CDN is the most basic optimization for distributing heavy assets like videos and images, which news websites use way more than text. Why not use a CDN?
So put them on a separate domain and serve that from a CDN if you really care whether that stock photo no one notices loads in 500 ms instead of 2000. That doesn't explain much why anyone would put their main domain behind cloudflare.
We have enough evil companies who invade our lives through the platform develop. Cloudflare is not one of them. Using them is voluntary (by the service providers), and I think they're one of the more useful companies around.
BTW, I'm not a web developer or Cloudflare employee. I have no skin in this stuff, however they build some cool stuff inside the Linux kernel, which is interesting from my PoV.
> It all gets loaded once and then cached in the browser.
Then cleared and/or invalidated by the user or browser's logic itself due to plethora of reasons.
> The initial load takes long regardless of whether there's a CDN.
Actually, no. A reasonably fast internet connection (>12 Mbps we can say) can load a lot of things very very fast. The biggest overhead is DNS, even with CDNs. With a good local, network-wide DNSMasq installation, if the server is close, I can load big sites almost instantly.
> And, it doesn't matter whether a JS file loads in 50 ms or 300 ms, if it then takes 5 seconds to parse and start running.
I think 5 seconds is long time for even the old Netscape Navigator's JS parser. You need to run something akin skynet to parse the JS file for straight 5 seconds. How's that even possible?
> So put them on a separate domain and serve that from a CDN if you really care whether that stock photo no one notices loads in 500 ms instead of 2000.
I don't know you, but world news generally contains live/new footage or fresh photos from the ground, not some stock photos, also we humans are visual animals. Many people want to see the images first, read the text later.
> That doesn't explain much why anyone would put their main domain behind cloudflare.
Load balancing, DDoS protection, CDN, workers, Bot/Scraping protection, cost reduction, rate limiting, you name it. Even my DSL router implements some of the protections, to my surprise.
Internet is not the same beast now when compared to 90s/00s. I miss the simpler times, but alas.
For me, the highest was 310ms round trip to Singapore, so higher than your estimate but not too bad.
But this is completely beside the point. As far as I know, if you're using a CDN effectively (i.e. a large proportion of requests are hitting cache), it should be cheaper than having all requests hit your server, not more expensive. So even if you don't "need" a CDN, you might want one. This is orthogonal to the issue of bot traffic, which exists whether or not you use a CDN. If you want to use a CDN but don't mind the costs of bot traffic, you can configure CloudFlare to not show the CAPTCHAs, or use a different CDN.
AWS does offer a CDN, right? Somehow they do it without captchas and without me ever noticing. So I'm somewhat right at cursing at cloudflare because it's the only one actually announcing its presence by actively disrupting your browsing.
My old business had users in countries around the world, and the assets were highly optimised for speed. However adding CloudFlare (a) significantly sped up our service to clients, especially those in Asian countries, and (b) significantly improved reliability of connections because CloudFlare have their own dedicated network links between countries and/or optimised for reliability.
One of the things using cloudflare gets you is all those protections without having to know how to do them yourself. Which a lot of the developers don't know how to do.
There's also something to be said for catching a lot of this at the network layer on a cluster of machines that can handle any incoming traffic that your one poor neglected vm can't.
If worried about a ddos attack cloudflare or another provider might be a good choice. But adding ddos support by default seems unncessary. In my 20 years of running 100s of sites I haven't run into a situation where I need ddos support. The vast majority will never be the target. Once in awhile google or bing will ddos you but using cloudflare to block that seems like overkill.
We might weight those tradeoffs differently. That's ok.
For any of my pages that do happen to use Cloudflare, I am luckily able to disable this discrimination in the CP so kudos for that at least, but terrible defaults imo.
Those are the two I find repeatedly blocked when accessing via Tor. The former by Cloudflare, the latter by Google.
I use Tor to lookup phone numbers that have just called me, to decide whether it's a good idea to answer. Since I don't want to be personally associated with such numbers I prefer to search anonymously. But often it's impossible to get a result.
Sometimes even spending 5 minutes solving captchas isn't enough. (I'd only spend that long to see if it's just an outlier. No, it's quite common.)
This creates an immense pressure to tell various services exactly who is phoning me, which is a terrible attitude to privacy.
It's not your choice if the site owners/admins use cloudflare. It IS your choice not to use those sites.
There is no "don't use" if I want to get my task done.
I can choose not to obtain the information, but then I still have the problem I started with.
tor has some absolutely valid and import use cases, but what percent of tor exit traffic is actually someone trying to keep their traffic anonymous from the eyes of an oppressive regime, and what percent are script kiddies, or someone hiding torrenting from their isp?
Not sure what "bad actors hitting a page" even means. I host public info so people can see it, be it good or "bad" people. Let them see it.
DDoS is different and can be devastating of course. Also, very rare. In decades hosting content (started my first hosting business in 1994) I've never experienced anything remotely like a DDos. I know it happens, but definitely very rare for most people. Driving tons of legitimate users away with relentless captcha annoyances for the once in a liftime possibility of a DDoS is not a good tradeoff.
If you're in a business that attracts DDoS like flies then deal with that, otherwise lay off the captchas.
I would refrain from commenting on this topic then.
> I host public info so people can see it, be it good or "bad" people. Let them see it.
If your site ever gets big enough, you'll understand.
I’m not sure whether this is true, but it seems like with Firefox I get these captchas much more often than with Chrome. Sometimes they’re so difficult to solve it really takes a minute or two to do so, and it’s incredibly disturbing / an unfriendly interaction.
Surely there must be a better way to deal with this? Why do I have to keep proving again and again and again to Cloudflare I am, in fact, a person?
I actually agree with your overall point as I try to use Tor for a lot of "normal" browsing, but I'm not sure what the correct solution to accommodate both is. It's a hard problem, and having been in that position myself I have a hard time faulting small website operators who have no alternative defenses.
e: just to add to this, I see the existence of ddosing as a significant driver towards centralized monolithic services. If your blog on Palestinian rights or whatever is getting hit, that's an incentive to move it to a platform that takes care of networking for you. It's a little absurd to go all-in on decentralized self-hosting without at least an acknowledgement that with current tech and typical personal-computing budgets, doing so is giving a heckler's veto to literally everyone. Cloudflare isn't the only dimension things can be centralized along.
Do you pay a variable amount for your hosting? How and why? All VDS and dedicated server offerings I've ever seen are fixed amount per month. And more often than not the network is limited by speed, not by data transfer.
Mind you, I see your point and I generally don't like the captchas either, but it is definitely a trade-off and I won't blame webmasters that use the DDoS protection.
One bot that's just stuck on a loop or being overly aggressive is going to have one IP.
Unless you're an original ARPANET contributor, there have always been attempts to control access and stop attacks. You're making the same mistake every conservative does. Longing for a nostalgia that never existed.
That's not what they said at all. They complained about breaking the access to things that were already chosen to be public.
If the key being shared is embedded in the device, even in a secure enclave or something, then my understanding was that would open the door for key extraction. If the key is unique per-device, then that's not a problem. But if the key is unique per-10,000 and stored statically on the device, then hacking one device means that key can be released to anyone and the entire pool can be imitated.
So if the above is correct, it can't be that a single private key shared across the entire company is stored on the device because that key would be getting constantly extracted and leaked by some determined hacker somewhere. But if it's a unique key per-device, then... I just can't figure out how validating that key wouldn't require transmitting unique information to somebody, whether it's Cloudflare or the device manufacturer.
Where am I going wrong? I feel like I'm misunderstanding something fundamental about how signing works on these devices, but I can't figure out what it is. If I buy a Yubikey, is it connecting to the manufacturer's servers and getting a new key each time it's used? I thought they worked offline.
Or are secure enclaves just much more secure than I think they are? Are we assuming that it's impossible to extract a private key from one of these devices?
Basically:
* Attestation keys are not unique per authenticator; they're shared among batches of authenticators.
* If you extract the batch's attestation key, you can imitate authenticators from that batch. That doesn't mean you can authenticate as a registered authenticator, of course; it just means you can pretend to be a "Yubico XYZ" device.
* Yes, I think Cloudflare is assuming it's hard to extract the attestation key. I think this is basically a safe assumption, but if it isn't, they can always choose to distrust batches known to be compromised.
- Does this mean if I buy 2 of these devices at the same time, it's possible for me to get the same attestation keys on both devices? I guess depends on how many batches at a time a company is producing.
- Doesn't this mean that attestation keys will get more unique over time as devices from the pool fall out of circulation and become rarer? Are keys rotated to prevent that (ie, would a manufacturer ever re-release a new pool with the same keys as an old one)?
- To my (limited) knowledge, yes, you are right that keys will get more unique over time. That's a very good point. Keys are not rotated nor (generally) are they rotatable; they are usually read-only. If you are using a very old FIDO device and worried it has too much entropy now--like, if it's a "rare" or "vintage" device!--then you should buy a new one, I guess?
(I honestly have not thought about your second point before, but I am not really deep in the FIDO stuff. So take my answer with a grain of salt.)
Such behavior would require some way of revoking old keys from newer devices to prevent a situation where a compromised and blacklisted old key is selected and causes the CAPTCHA to fail, seemingly at random.
But of course, despite this, still a unique key is generated for each identity upon sign up [0]. I am not sure (as in 'have no knowledge of') the entropy for these devices.
[0] https://fidoalliance.org/fido-technotes-the-truth-about-atte...
Which effectively means bricking the devices of 9999 innocent users each time.
Why are we creating a world where users will be told they can't visit a website or access their account any more because they didn't spend enough money on a hardware DRM device which tries to hide a key from them?
Vanguard restricts (or used to) trusted keys to those made by Yubico. This annoys me, given they let me use any phone number. So this suggests (as you seem to be arguing) that attestation is an attractive nuisance—when you can't tell whether a phone number is with a trustworthy or untrustworthy exchange, teleco, or VOIP service, you just have no choice but to trust it. But when you can enforce attestation, some Vanguard employee with a Yubico Certified Security Engineer certificate (I made that up) decides to restrict us to Yubico's keys.
On the other hand, FIDO aims to also support enterprise usage, and I think it'd be a nonstarter to say that enterprises shouldn't be able to restrict you to officially issued authenticators—and without remote attestation, they'd have to do more cumbersome steps like handling enrollment centrally (and then shipping enrolled authenticators to users), which, in a distributed workforce, sucks.
So attestation fills a need, and I think we just have to hope it's not widely used or abused.
Is Cloudflare's usage an abuse? Eh, I don't know. It's an interesting experiment.
It seems to me that this use case is completely at odds with the general use cases we expect when people browse the web.
In the enterprise, society accepts the idea that someone's employer knows their name, their address, their bank details, and even everything they browse on their company-issued computer. Whether that's good or bad, it's not a model that FIDO should be trying to support.
This reminds me of the controversy around TLS 1.3 allowing forward secrecy.[0] If companies want to make their own alternative to FIDO which has DRM and user tracking, to control access to their intranets, that's fine, but we shouldn't have allowed FIDO to become a Trojan Horse (or Trojan Dog?) to sneak this anti-feature into the general web.
[0] https://www.eff.org/deeplinks/2019/02/ets-isnt-tls-and-you-s...
Let's imagine I'm a consumer bank. I want the following features:
* I ship users a FIDO key when they sign up for an account.
* Users can register additional keys they bought or already have, and can use my key elsewhere.
* I consider malware on the user's computer to be in the scope of my threat model, and so I want all risky transactions (like transferring money) to come with a secure test of user presence.
I believe that the above three constraints are sufficient to motivate the current design. Why?
Well, let's say we do the obvious and just eliminate attestation. If the design otherwise remains unchanged, there's nothing to prevent clever malware from piggybacking a legitimate user presence tap to add a malicious, software-only authenticator to the user's account, which can then approve future transactions.[1]
Currently, however, I can simply require that authenticators all be non-self-signed, restricting to those that come with real hardware-backed security.
So now I have a few options. I can:
a) Disallow enrolling new authenticators. I think this is the worst option, because it means that the FIDO ecosystem would basically revert to (at worst) "an authenticator per relying party", which is expressly something it aims to avoid. (Think, keyrings full of USB fobs!)
b) Declare malware out of scope. I find this unsatisfying, since it suggests we might as well just replace FIDO with USB keys with X509 client certs, but see my footnote [1]--maybe at the limit these are equivalent?
c) Do what the FIDO Alliance seem to have done: create batch attestation, but discourage RPs from using it spuriously.
I think (c) is fairly reasonable, but as I said, I do find the entire malware argument sort of unsatisfying in a way, so, shrug.
[1] Arguably, if this is an attack you are concerned about, malware can also just piggyback presence to directly trigger transfers and other abuse, of course, so maybe this argument is a weak one? But I think an RP might reasonably want to say that "malware can do anything malicious within 10 seconds of a tap on your key, but if you think someone might have hacked your account, just remove your key and call customer support." If malware can add new software-only authenticators, it violates this functionality.
And presumably RPs in practice achieve this by subscribing to a list of key updates managed by the FIDO Alliance themselves. That seems like it puts a lot of control over the web into the hands of a group whose incentives may not be the same as those of the average web user.
As a point of comparison, my impression is that it's been quite difficult to get all Certificate Authorities to correctly follow the CA/Browser Forum rules, and to tighten those rules. That forum at least tries to balance the wishes of User Agents against those of the entities selling (access to) the keys, which I'm not sure if FIDO will achieve.
> malware can do anything malicious within 10 seconds of a tap on your key
If this is the security situation that FIDO aims to create, then I'm not sure if requiring a whitelisted hardware device adds anything, relative to allowing self-signed software-only authenticators, other than security theatre and a single point of failure (namely the whitelist itself).
I'm not aware of the FIDO Alliance doing this today. I guess anyone could create such a list, but while I think your concern is valid, it has not (yet) played out that way, so I don't think you can claim it's inevitable.
Anyway, I laid out what I believe are the reasonable alternatives. I think you're taking the position that local malware is already sufficiently powerful that it's not meaningful to restrict adding new authenticators?
I entertained this as well, and I think there's validity to it--but I do think it's reasonable for an RP to say, "If you unplug your FIDO key, the attacker is thwarted", which is not something that remains true if you allow self-signed authenticators.
Nope, it's just "expensive" to extract keys from secure hardware like this. The problem with an approach like this is that when the secret keys are identical for a large number of devices then the cost of revoking a compromised key goes up significantly which, for a spammer, would increase the value of obtaining said key because of the likelyhood that the key would be usable for a much longer time than if the key was unique to each device (and could be very easily blacklisted).
Techniques to extract data from these sorts of secure devices include various forms of side-channel analysis, decapping and microprobing the IC, using SEM, etc. to physically damage parts of the circuit to try to force it to disclose the key and various forms of power and clock glitching.
Most decent hardware-based cryptosystems are designed to ensure that each device has a unique key so that the cost of extracting one key (lets say around $100k) is too high for a potential attacker if the key can just be blacklisted, but if the key is expensive/impossible to blacklist then that cost might be worthwhile to an attacker.
Whilst Fast Identity Online (FIDO) is much more than WebAuthn, Cloudflare's proposal here is to use WebAuthn to get rid of CAPTCHAs. The official WebAuthn doc is surprisingly accessible with neat illustrations for key topics: https://w3c.github.io/webauthn/ (ref registration and authentication ceremonies, in particular)
This provides effective rate limiting and you can still get every key you automate banned very easily.
For a major sporting event, one of our sites was heavily targeted by “free TV streaming services” self promoting their stuff.
No amount of Google CAPTCHA or Cloudflare could stop it while keeping it online. Never seen anything like it in my life.
I HATE CAPTCHA's with a passion. They are everywhere and constantly slow me down. And you mentioned, they are likely not helpful in stopping bots.
That's why. Try using Firefox without being logged in to Google and with an ad blocker - you really won't like it.
In theory, couldn’t Yubikey begin reducing batch sizes to 1,000 and Cloudflare mark specific batch numbers as requiring one extra step to verify? The vast majority of Yubikey sales will be for real people in any case.
Most of the abuse that CloudFlare protects from is also usually illegal. And those taking risk of doing something illegal usually do it because it's highly profitable so making some authentification devices a bit more expensive won't make any difference.
"In this Full Basic Attestation model, a large number of authenticators must share the same Attestation certificate and Attestation Private Key in order to provide non-linkability (see Protocol Core Design Considerations). Authenticators can only be identified on a production batch level or an AAID level by their Attestation Certificate, and not individually. A large number of authenticators sharing the same Attestation Certificate provides better privacy, but also makes the related private key a more attractive attack target."
It's basically the same both ways. You don't trust them with your private info. They don't trust you, either. The easiest way is, indeed, to just call the whole thing off.
Everybody would love an alternative that lets more get done with less trust. Sometimes they find them, for limited cases. But nobody's solved it for the general case.
You may have noticed that both Facebook and Google already use cookies. Did you know Hacker News has a cookie too?
There's a difference between being logged in to an account on a single site, and letting Big Tech track you across most of the Internet.
You might be thinking "Duh, they can ask the Security Key". Nope, the Security Key hasn't the faintest idea, remembering that ID is the Relying Party's job, the Security Key isn't going to remember.
I feel like I'm crazy – no one else complains. I've mentioned @GumRoad on twitter but nothing.
I think reCAPTCHA just flat out breaks any form on macOS Safari. I've had a few work, so I'm guessing they changed something and it requires special configuration by the website using reCAPTCHA for macOS Safari to work (but apparently no one does it). I can't even login to Gumroad for example. I know macOS market share is small, but damn, this has to be costing businesses money.
--------------
Okay I finally researched this. It looks like it's because 'strict-dynamic' isn't supported. https://bugs.webkit.org/show_bug.cgi?id=184031.
I can't imagine how many people have switched to Firefox/Chrome because of this issue. You basically can't register for anything, since reCAPTCHA is used practically everywhere.
Can we make CAPTCHA obsolete with decent micropayments solution, when you pay for every transaction with every website, just like we pay for every drop of water we use? Perhaps ISPs could handle it for us?
Sure, you are paying for the every drop of water, but what if you really wanted to pay for water from a specific region, doesn't want water from another region, and trust that the water company does not keep a cut or rip off either of you?
Anti-bot measures are also used to try to prevent password guessing on e.g the login site to gmail.
Finally, sometimes some places offer things like tickets that go very quickly, in which case having a bot reload the page means the tickets are likely to go to somebody owning a bot rather than a fan of the performer.
None of these cases are solved by payments, they are solved by client side certificates and, in the last case, by requiring the name of the people who are to use the ticket.
From a technical point of view it is possible. Assuming that the payments were mediated by some party that party could issues statements like "this user has used their monthly allowance but they would pay". Assuming that this provider is widely trusted websites may treat this as a "real user" and allow the visit. (This is roughly how https://coil.com/ works.) Of course there are negative implications such as making it very difficult for new providers to get started.
You can also imagine some type of smart-contract where the subscription fee is split at the end of the day or month amongst the visited sites. Upon visit the sites just get a token for one share. Of course this would need to be very carefully designed to prevent abuse. (Example one malicious client splitting their subscription across millions of pages)
Unfortunately corporations are not good at going a step back if the step forward is good for their business.
Tech monopolies have always had a vested interest in locking up user data, dictating the policies, and enforcing their own ownership rights. It used to be that only the largest and most sophisticated companies had the resources to shield that data, but Cloudflare changed all that. Walls are now trivial to set up, and virtually unbreachable, and that has forever changed the character of the internet by enforcing monopolistic policies with such technical precision that they're virtually impossible to overcome.
The ‘Internet 3.0’ isn’t coming because of Cloudflare. It’s coming because these monolith big tech companies have an army of engineers who have been centralizing and building it this way for years.
Cloudflare didn’t build these walls, it’s more of a giant boat now navigating it because other companies have no choice.
I like to think of them as giant data ferryman in this regard versus “a wall builder”.
I’m not saying frustrations aren’t warranted but — like come on — have a little perspective of what’s really happening with the Internet and who is actually driving it.
GDPR-like policies will continue to flood as governments partition their Internets and data making it harder and harder to run international Internet businesses.
I'm not particularly happy about things either (especially crawling access), but it will be a net positive whenever you can level the playing field with competition.
When the biggest infringers of data are driving the creation of government policies that only they can circumvent and navigate -- that's a serious, serious problem.
Plus there is already an endless sea of walled stuff (paywall, "create an account and login" wall, "fill in your address/ZIP code and we'll give you the best price" wall, government filtering wall).
I'm not saying it's great, or it would be great, but currently having 1 (or maybe 1-and-half) search engines is also very far from ideal.
Sure, bots can automate keys, but those keys could also be banned just as well. Cloudflare only needs to know which ones are the good keys and track those forever. This means, for every non-bot out there, the CAPTCHAs are as good as gone.
The genius of Cloudflare here is that they (ab)use WebAuthn, which can also be implemented on Android and iOS natively. Before you know it, Cloudflare has built an identity platform (where it may not be helpful for KYC) is plenty useful for websites Cloudflare fronts. Imagine never having to bother with user registration and authentication and bots... that's the next extension I see to all of this.
It's an interesting trade-off. It seems like batch keys for device attestation was designed to help protect individual privacy (good), but if you can't ban a key without potentially a lot of splash damage when you detect a bad actor, that seems like a very limiting choice.
Maybe it could make sense for a bank to do this, sending account holders a special custom Security Key with the bank's branding on it. I personally think that's stupid, but I can imagine it appealing to bank executives and it's not so stupid as to be worse than SMS or TOTP 2FA that banks do today.
But it clearly isn't relevant for no-cost services like Facebook or Gmail, and so sure enough you can just tell them you don't want to give them attestation and they work anyway (I don't know if either of them ask, I just reflexively deny attestation if it's requested).
It isn't intended to be useful for trying to do stuff like Cloudflare are attempting here. Which doesn't mean Cloudflare can't succeed in their goals, but in the FIDO threat models a "bad actor" would be a whole vendor, maybe some outfit is using fixed long term secret keys inside their Security Key products and they just sell the NSA a list of those keys - you might decide to just refuse all the products from this vendor. Whereas for Cloudflare the "bad actor" they're worried about just buys a half dozen of whatever was cheapest from eBay and then plugs them into a Raspberry Pi.
Or, do they? That's the gamble I think Cloudflare is taking. Maybe the value of defeating this intervention is so low that bad guys will not, in fact, build a Raspberry Pi Security Key clicker proxy to make their thing work.
You're right. I meant Cloudflare could ban the generated public-key and not the device's public-key itself. Besides, they could also mark the batch as being taken over by bots and increase the level on challenges issued to the batch. Note though, a single secure module can only generate / store so many public-keys. For instance, Yubi Key 5 supports up to 25 keys, though those could be reset to generate a newer set of 25, but repeat registration of a number of keys from a single batch is bound to trigger some statistical anomalies.
From Cloudflare's blog about Cryptographic attestation of personhood https://archive.is/4EbER
> For our challenge, we leverage the WebAuthn registration process. It has been designed to perform multiple authentications, which we do not have a use for. Therefore, we do assign the same constant value to the required username field. It protects users from deanonymization.
Currently, the user-name field is constant for all users. I wanted to point out that that they could amend the registration ceremony to register any user in particular.
This is for resident keys. A YubiKey 5 supports an infinite number of non-resident WebAuthn keys, because the returned key handle will simply be the private key encrypted with a master key stored on the YubiKey. For authentication the service will send the stored key handle back to the YubiKey which then can decrypt it and use the decrypted private key to sign the challenge.
Envelope encryption. Neat. Can WebAuthn keys be (made) a resident key? If so, is that preferred instead?
Conversely, what use case is there for resident keys in context of WebAuthn? For example, if there are multiple master keys, can I switch between them per browser / website (assuming the master key itself is a resident key and not burnt into the element)? Thanks.
The website will need to support this of course. Also the amount of storage available for resident keys on the YubiKey is limited.
I also thought the idea of the key exchange being fast was a red herring. That's a bad thing. If I'm them I'm paying attention to how long from prompt to exchange it takes a human to touch the button. On my own setup my key is on my laptop, which is in a dock. I must stand up and tap it over my monitor. It's just a few seconds but it's a) consistent in timing b) not < 1s. Imagine the aggregate timing data they have.
Overall they make some good points if you are teeny tiny player and ignore completely the scale cloudflare is operating at.
Google must be docking points from websites that employ captcha then, right?
(Partially sarcasm, I do know that Google does do some anti-cloaking crawling)
“Yesterday, the bankruptcy court approved the sale over the objections of several parties, including the Federal Trade Commission (FTC) and third party manufacturers Apple and AT&T who sold products to the bankrupt retailers.
...
The FTC’s objection was made to the court-appointed consumer privacy ombudsman in the RadioShack bankruptcy. Specifically, the FTC’s letter alleged the sale of personal information constitutes a deceptive practice because in its privacy policy, RadioShack promised never to share the customer’s personal information with third parties.”
https://www.jdsupra.com/legalnews/radioshack-bankruptcy-cour...
In that case the judge allowed the sale of the information in contradiction to its commitments.
Note that bankruptcy always allows things in contradiction to commitments; bankruptcy is all about balancing which commitments will not be fulfilled, and by how much, when a party is no longer capable of fulfilling all of its commitments.
If you don’t like particular commitments being voided in bankruptcy, you want legislation specifically protecting them so that there is a clear legal barrier to voiding those specific kinds of obligations.
Great in that they have a fantastic UI to add your site in, basically shielding the average user from attacks.
Bad from a standpoint of that now only Google, Bing, and maybe other big search engines have the capabilities to actually crawl the internet now.
I don't see us getting a massive innovation in search on the internet now that Google has such a massive foothold, and companies like Cloudflare stop innovation from happening.
How are we "stopping search innovation"?
But let's suppose someone is building a new cool search engine and our ML stuff is blocking them. Then... contact us/me.
(but Google is allowed because Google was first to market)
Yes, you must adhere to the controls that site administrators put in place, like Cloudflare.... You don't get to blast my site with requests, just because you want to...
(b) But you’re a-ok with Google doing this. Gated communities aren’t really good for anybody but I see what you are saying.
The same is true online. Apple's walled garden has kept hundreds of millions of people safe on their device. It's why iOS malware isn't a thing.
> Cloudflare stops much more than just blasts
Exactly. There's even more benefit to Cloudflare than just DDoS. Captcha's for stopping credential stuffing, for example.
But hey if I pay Cloudflare enough, then I’ll get to blast your site and possibly stuff creds at the same time :/
[1] - https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...
So no problem, time to move on web search is no longer exciting
How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl?
Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.
And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
They're only a gatekeeper because sites voluntarily enter into commercial agreements with them. There's no coercion or manipulation like Google AMP.
When I had a site that had millions of pages, I found that sites like Baidu would crawl my site as often, if not more often than Google.
I already felt the relationship with Google was parasitic, but I looked through my logs and never found a single hit that came from Baidu and many of the other search engines that would overload my site.
I was looking at a substantial part of the site running costs going to supporting web crawlers that were not doing anything (1) to help me, or (2) to help end users (if they don't want to send Chinese users to an English-speaking web site, why crawl the site?)
So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition.
There are web crawler behaviors that are annoying: harvesting email addresses, overloading your site, etc. But how do you know who is doing something wrong with the data and who is just collecting it do do nothing with it? (Probably 95% of web crawling ex. Google.)
This sounds like you're onto a reasonable “legit” factor: does the crawler honor robots.txt? Baidu would be legit because they don't lie about their identity and if you put a rule in your robots.txt file they'll honor it.
Yes, I've never understood why it's seemingly so important to CAPTCHA me before serving me less than 100kb of read only plain jane HTML. What sort of "attack" is this stopping? I'm pretty sure the CAPTCHA itself is bigger than half the sites it blocks me from reading.
Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
Which makes "contact us for an exception" a no-go, as the relevant source IPs will constantly be changing.
Recently your company announced partnership with Internet Archive, but if CloudFlare want to continue play a role as behevolent party everyone should have equal access to this data. Yeah it means that some bad actors will be able to easily scrap the web too, but...
CloudFlare service can't prevent scrapping anyway. There are shady residential proxy networks, services to bypass captcha and scrapping software like Zennoposter. It's possible to make scrapping more expensive, but bad actors don't care because they have money. Unfortunately enthusiasts, open source projects and small companies don't have enough resources to do the same.
I think you might not understand that it's site owners like me who want to stop scraping. It usually comes from specific bad incidents, like copycat sites stealing our content and work.
Cloudflare wouldn't block scraping if website owners didn't want it. And website owners can easily disable this protection.
Also I'm talking from experience because I been on both sides of the fence: doing scrapping and implementing protection. So yeah your CloudFlare protection will deter 10% of bad actors, but will also cut off 99% of enthusiast / research efforts or users of niche software or browsers. Still anyone with $1000+ budget will scrap whatever they want.
If this doesn't at-least meet your definition of "reach a human at Cloudflare", I'm not sure what will.
That said, I don't think it's a good situation that this is the solution rather than a proper, documented position that people can work to.
In fact it's a textbook definition of a non-scaling solution.
Individual companies having so much power gives me the willies.
Thanks for taking the time to reply.
You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when Cloudflare suddenly decides it does not want to honour that "agreement"? What happens if/when Cloudflare is sold, or the contact who greenlit these smaller "legit" crawlers moves on and decides that it no longer agrees with said website anymore?
Is a price comparison site a "legit" crawler? What defines a "bot" vs a "crawler" in the eyes of Cloudflare?
Would you need to notify your customers that you now also allow additional crawlers access to their sites, or would they need to opt into it via the Cloudflare dashboard? What happens when you have a falling out with my said company (it happens, relationships sour) and suddenly we can't make contact, then suddenly customers websites aren't being scraped because we're bots?
If you are a price comparison site getting blocked by cloudflare, site owners may be losing sales, and that's good feedback.
Depending on the industry, etc..
I really can't think of a good solution. But that's the tricky position Cloudflare is in - how does it balance everything.
No such filters should exist. Is it really that awfully bad without anything but basic filters (ban an IP for flooding)? Are there like, operations that try and spam every single Cloudflare-hosted website 24/7?
Legitimately curious if your anti-bot measures come from actual bad experience with the internet or is it just a liability limitation move? (Namely to reduce potential suing surface by angry data owners and/or three-letter agencies.)
---
Basically, if I am experimenting with a basic crawling program and I hit websites A and B 20 times each in a space of one hour, is that really deserving of a captcha or extra auth methods?
Not flaming but I am really curious. Do you have any data and rationale posted somewhere that go into deeper detail about why Cloudflare's bot detection is how it is?
That isn't to say that the customers are savvy and have a good understanding of different types of automated traffic and which automated traffic is harmful and which is benign. Many have a quite naive understanding that doesn't extend beyond "bots = bad, unless it's Google" and dial protection settings to the max for no good reason.
It turns out that he was right, sort of. The vanilla attach server into the internet, server-to-client IP-network is pretty much dead. It has been replaced with CDN's , private delivery networks, cache on top of cache. Cloudfare, Amazon, Google and MS are the connection points for the IP. Their internal network infrastructure transfer most of the data.
The fact that 98% of traffic goes trough this new infrastructure, allows some still plug their server to the net raw and their traffic still gets trough.
Doesn't mean that 98% of the value of the Internet results from this traffic.
Even if you discount all of the web, there still lots of applications using the federated model (e.g. SMTP) or peer-to-peer (e.g. Crypto, VOIP), that require end-to-end connectivity.
My original point is about capacity and if the old internet could work today. It seems like I'm correct, but I'm not so sure. I would like to see other opinions.
Every response so far is "there exists". The real issue is if the internet could do everything without caching data near the edge.
Terrestrial optical networks operate far, far below capacity to create artificial scarcity, which is to a certain extent necessary to recoup the capital expenditures in a competitive market, and is to a certain extent an abuse of an under-regulated natural monopoly.
If you could eliminate all adversarial factors and put every data service subscriber's payment for a single month into a pool, and that pool purchased only transceivers, passive optics and switches, and this hardware was distributed to every network operator perfectly fairly based on its contribution to the global maximization of available network capacity, then the delivered capacity to the end user could increase by something like 4.5 orders of magnitude with no substantial change in topology or subscriber or provider costs afterwards using the existing fiber, with a few more orders of magnitude possible with a fatter tree before the backbone costs explode.
With DWDM you can carry 100+ channels of 100Gbps over a single fiber today, with commodity, off the shelf components. Most fiber in the ground today is probably still lit with a single wave of 10G, if it's not just dark.
This distribution model is not even remotely a technical necessity, it's an arbitrary local minima reached largely by exploitative market distortions and adversarial economics.
It's just like Windows - just because 95% of the Internet does something one way doesn't mean it doesn't suck, isn't more complicated than it needs to be, and doesn't cost more than it needs to cost.
Anyone with a little bit of bandwidth and a Raspberry Pi can run a web server, even with dynamic content.
1. Filtering ever increasing trillions of spam/clickbait pages
2. Figuring out which results are useful information vs corporates trying to sell something.
Those problems are not solveable by a couple guys in a garage
Reminder: Google Is An Ad Company. Are you sure they actually want to solve search? Their primary interest is to be that corporation selling you something.
With those two considerations, this actually seems like a really good idea to me.
IdPs like Apple/Google/Microsoft might do a fine job of limiting you to "one account per $unit-of-hardware"; Apple in particular can do this via iOS attestation. But then you're limited to either their heuristics (in the case of MSFT/Google) or their hardware (in the case of Apple).
Ultimately this is apples-to-oranges, though, since Cloudflare is not offering an IdP product but simply an anti-automation solution. If you use federated auth, you're getting (and giving up) a lot of other stuff beyond just anti-automation.
I personally enjoy the ability to use my employer-provided authenticators on my personal accounts. I think this is both convenient (only one thing to carry) and helps adoption (I got a key “for free”). So maybe enterprise use is a Trojan Horse to promote consumer adoption?
Of course this idea raises privacy and authority concerns, similar to certificate authorities.
I think that would require that every key maker would have staff in every city in the world who were trained to inspect your identity documents, check for forgeries, and not be susceptible to bribes or coercion.
Or at the very least every city would have to contain at least one location where someone from some organisation (possibly the government) could carry out this process.
The "authority" concern would require that the verifying organisation could be blacklisted if they started giving out too many IDs to the wrong people, or refusing to give IDs to the right people. (Perhaps you've played the game "Papers, Please".)
It's the "privacy" concern that worries me more. What happens if someone tries to buy a second key? Presumably there is a limit to how many each person can buy, so if someone says they lost their previous key, the issuer needs a way of revoking it. But how do they do that without creating a valuable list somewhere of which key belongs to which real world ID?
To make matters worse, because people can move between cities and jurisdictions, this database of online IDs would need to be globally shared between all verifiers, otherwise you could just buy multiple IDs from multiple vendors. That means the database that ties your browsing history to your legal identity will be accessible to basically everyone in the world, because the access to that database will only be as secure as its weakest link.
If we're going down this route, we might as well put everyone's identity onto a blockchain and let people vouch for each other in order to establish trust/reputation. In fact, that sort of system has actually been implemented; it's called BrightID, and it "requires no personal information, letting you prove your humanness without risking your privacy".
Or is there some repository of all allowed devices with identifiers? Intuitively that'd be the only way to prevent infinite virtual devices..
If so, what if the secrets get out? Then all the keys in those batches are poisoned?
Isn't this just some sort of side channel certificate authority?
I work with Cloudflare sites and it is clear that thier current enterprise offerings are hard to tweak to solve attacks without spamming the users with captchas. The captchas are already too complicated for the average user so it is mostly turned off even though it has other consequences. I have to look into this new thing though.
As much as I like the idea of an open Internet to be used by anyone from anywhere. It does simply not work today for a lot of enterprises.
The open Internet is already dead thanks to Cloudflare, Akamai etc. A lot of European companies use theses services to block China, Russia, TOR, VPN-services and so on.
I think the nature of the internet necessitates temporary, limited gatekeepers like CloudFlare, but as long as the legal status and technology keeps maintaining moving targets (like how you can fake your way out with a dedicated ip vpn, or just remain within tor network) this isn't too much of an issue. Anonimity+trust will never be a fully solved problem, and as long as a comfortable level is achievable between willing parties, the status quo remains.
This cannot possibly end well.
[1]: https://blog.cloudflare.com/introducing-cryptographic-attest...
Physical keys (and biometrics) work well because they are rarely called for, and the user knows they are doing something security sensitive. "This random page asked me to insert my security key" can't be healthy.
If any page can request any other pages response that'd make the whole system pointless
What the user is doing is not security sensitive. They are, in fact, themselves, and that's all the Security Key is confirming. "Yup, still me".
One of the easy ways fools trip themselves up here is that they think this is identifying information. But it isn't. "Yup, still me" doesn't identify anyone. The identity was already known to your interlocutor, which is why "Yup, still me" is enough.
And that's what's so clever about the FIDO design. A Security Key has no idea who it "is", it just knows it's still the same as before. If you're already authenticated as Jim Smith, you can enroll a security key "Yup, still me" -> the Relying Party stores the information, and then you can later sign in using it to verify your identity, "I'm Jim Smith". "Is this still you Jim Smith?" "Yup, still me".
So that's why this doesn't help bad guys. "Are you still er... you?" "Yup, still me". Completely useless. Of course you are, that doesn't help them at all.
(disclosure: work there)
This sort of "we can solve that problem; we just need to kill your privacy" seems to be par-for-the-course in SV-style companies.
I really wonder if anyone involved with building these systems has ever seriously thought about what could happen if the data collected (or that could be collected) by these systems was obtained by an adversary.
Not to mention the incredible incentive problems that are created by designing things that are designed in a way that _requires_ that individuals are tracked across the internet.
I know that CloudFlare is just one of many companies that is moving in this direction and they're certainly not the worst offenders when it comes to slowly murdering individual privacy (Facebook and Google are obviously far worse) but they have a uniquely powerful position due to the number of sites that use their DDoS protection and seem to be taking a casual disregard to the damage that they can do to people's privacy.
I know for a fact that there are staff at CloudFlare who are aware of this problem but nothing has changed, so I guess that they don't care that they are making some sites unavailable to anyone who has to use Tor.
(although practically I'm unsure as to whether that's really a good idea or would work well)
Meaning you need to buy more. Makes it expensive at least.
The site records the ID and public key.
When you return, to confirm it's really you, the site sends one or more IDs you've enrolled and says, sign this fresh random data with one of the associated private keys.
Your tokens can look at the site and an ID and decide if they made that ID for that site, if they did they sign the message with the private key, proving you are still you. If they didn't make it, they pass and maybe you own a different token that can sign, or maybe you show them a different ID they do recognise.
To reuse this capability for tracking, the site would need to guess who you are first. "I guess this is arsome, they have this U2F key". But if they can guess who you are, they already don't need such tracking.
Edit: I was wrong. Cloudflare claims they could track people, but it would require tracking via cookies. [1] The hardware security keys have an "attestation key pair" that is shared among all units in one production batch (which contains at least 100K units). [2]
1: https://blog.cloudflare.com/introducing-cryptographic-attest...
2: https://www.w3.org/TR/webauthn-2/#sctn-attestation-privacy