Hacker News Reader

Comment by quotemstr | Hacker News Reader

Web PKI is cursed. Why should anyone have to ship a new version of Go, Electron, webpki-roots, etc. (possibly over a now-compromised TLS connection!) just to revoke a certificate? Why are so many packages independently shipping high-sensitivity trust-bootstrap credentials? And why can any CA still vouch for any domain instead of each being limited to a specific ccTLD or other non-total shard of subject names?

Also, it's quietly, in the email space, that we're seeing real innovation while browser vendors just say "no" to everything. We should be distributing trust, not centralizing it in browser vendor security teams and Let's Encrypt.

View on HN

Web PKI doesn't mandate that every client ship with its own root certs. The clients you listed can be configured to use a centrally managed trust store, if that's what you want. The fact that they don't by default is a deliberate tradeoff that the clients made, not some standard's fault.

This isn't, like, one misbehaving client. It's the way the ecosystem has evolved. GP didn't specify whether they were criticizing the standard or the ecosystem more broadly, but even if they were criticizing the standard, it's pretty fair to evaluate a standard at least partly based on how it gets implemented in practice. There's a reason cryptography tools for the last decade or more have been focused on being misuse-resistant.

Yes. I am criticizing the de facto web PKI, not the somewhat better design intent. The design intent is still bad though: we should never have given CAs universal issuance privileges.

The web has continued to go the wrong way in practice. (For example, browser makers rejected RFC9102 DNSSEC stapling, which could have enabled an incremental migration away from CA-centric hell.) Everything fix the existing web PKI community makes just seems to centralize trust more.

For example, Let's Encrypt has been useful for moving people off unencrypted services, but side it's now ubiquitous on the internet and had to work within the broken universal-grant CA model, it now represents a terrifying central point of failure. Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.

> The design intent is still bad though: we should never have given CAs universal issuance privileges.

This is sort of right and sort of wrong. It's true that you could have a DNSSEC-style regime where only one entity could issue for a given domain, but as long as you're going to allow third party issuance at all, then you have to contend with the fact that most CAs want to issue in the big gTLDs (.com, .org, etc.) and so narrowing issuance doesn't help that much in many cases.

> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.

This is ahistorical. By the time that DANE was designed, the WebPKI model was well entrenched and there was a huge population of browsers that couldn't really be upgraded and also only supported the WebPKI. If you wanted something that was acceptable to those browsers, it had to be WebPKI and it had to be cross-signed, hence Let's Encrypt.

I suppose you could argue that at some point after that one could have tried to deploy DANE and phase out LE and the rest of the WebPKI. I don't think that's true for the reasons that have been discussed on HN about DNSSEC, but that's really not the same as you wouldn't have needed LE in the first place.

Had we been using DANE instead of WebPKI, the response to this sort of compromise would be "aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem that there's no way to route around this DNS provider being untrustworthy."

> aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem

You mean until the DNSsec key rollover completes and they're no longer in that role?

As opposed to what we have now, an indefinite period where crappy non-updated devices will still accept these certs and nothing can be done about it?

The Web PKI works great for browsers. Everything else, not so much.

(To be fair, the crappy devices¹ would probably fail to deal with the periodic DNSsec root key rollovers. But at least that's a fail-closed.)

¹ a label that includes older versions of systemd-resolved, for whatever that may mean

DNSSEC key rollovers don't change who has custody of the keys. They're not a magic spell that fixes stuff like this. It is closer to the truth that DNSSEC is limited to recovery from incidents like this to doing rollovers, rather than immediate revocations. Rollover is not something that makes DANE more attractive here.

I don't think I understand what you're trying to argue here; if you can't find a usable registrar for your domain… you can't really use that domain anyway, can you? They can just break it. But… let's not get into yet another DNSSEC/DANE discussion.

I'm not sure I see what this has to do with how the DNS operates, beyond observing that key rollovers are not a reasonable response to incidents like this.

I'm not connecting at all on this discussion because I don't understand what entity/role in DNS/DANE is the equivalent to eMudrah that is to be distrusted and can't / has to deal with compromised key material. It could be either the registrar or the TLD operator. The former you can switch to another one. The latter just makes the entire TLD unusable no matter how you spin it and whether you use DANE or not. Is there some third role you're considering here?

This seems simple. Nobody is switching domain names as a result of this eMudhra thing; they're just going to switch CAs, when eMudhra is murdered by the root programs. But switching domain names would be the only thing you could do if a DANE key was mishandled.

I assume you meant DNSsec key, not "DANE key"? (No such thing, anyway.) For this discussion to make sense, it has to be the TLD's DNSsec ZSK or KSK.

The fix in either case is key rollover; if it's the KSK it needs to go to the root which will be annoying but doable. The TLD needs to do this, and it'll be complete, for everyone, regardless of updates or shitty devices, after the relevant TTL has passed. The TTL on the root zone's DS records seems to be 1 day.

If the TLD fails to do the key rollover for whatever reason (incompetence, maliciousness, …) - DNSsec is now broken on the entire TLD. So, security of DNS fails down to classic DNS with all spoofing problems and randomization/uniqueness countermeasures we've accumulated over 35 years. Their DNSsec signatures are still valid, DANE is still green. It really depends on your stance on this; personally speaking I'd consider the TLD unusable now and end the post here, but I assume you don't share that view.

Can you spoof/break DANE now? Yes, if you can break DNS. But in that case, you're in a whole lot of trouble already; if you can really break DNS (poison arbitrary resolvers) you can now get a 'classic' (non-DANE) TLS certificate from your favorite ACME provider. If you can't fully break DNS, that essentially leaves MITM situations, i.e. specific individual situations. Part of that is getting improved with DoT, DoH and DoQ (which, yes, have their own issues.) Realistically, browsers would be marking the entire TLD DANE-untrustworthy now.

On Earth-666, you could presumably roll out DLV [DNSsec Lookaside Validation] now — that's the "side root" mechanism that was used before DNSsec became ubiquitous at the TLD level. Most DNS people absolutely hate it. Ignoring that, the one TLD is now screwed and it's a layer 9 fix. Is that better or worse than a compromised CA resulting in every single TLS service being screwed? Neither, just different, I'd argue.

(As batshit crazy as it is, maybe DLV is the answer to the kind of compromise you have in mind. You seem to be happy with browser vendors having a lot of authority and control over trust. A browser could unilaterally decide to roll out DLV on a given TLD, or any level of domain really. Somebody has to run that as a service, though. Unless they ship the records with the browser… but that's getting really extreme now.)

For the sake of useful discussion, another case to consider is a malicious TLD that specifically hates you. (Or was given reason to hate you by legal means.) They can publish completely bogus data for your domain, including DANE records. But they can also get ACME certs… so I'd call that a wash. (Sure, they wouldn't be able to get OV certs. But a situation with OV certs is IMHO also a situation where you wouldn't use DANE exclusively, rather you'd require both DANE & classic PKI.)

I don't understand what any of this has to do with the point I made. eMudhra today announced that it's exiting the CA business entirely. They're doing that because the root programs were going to evict (and humiliate) them anyways. Nothing like that will ever happen in the DNSSEC PKI.

> I don't understand what any of this has to do with the point I made.

That's because I don't understand what point you're trying to make. Something here is probably blatantly obvious to you and entirely opaque to me.

> Nothing like that will ever happen in the DNSSEC PKI.

This is a nonsensical statement to me. What is "like that" in DNSSEC? A TLD operator fucking up badly? That's not even the same class of event since the blast radius is at least limited to their TLD(s). For most TLDs (though not gTLDs), the layer 9 "owner" isn't the operator, it'd result in that contract getting cancelled and someone else running the TLD instead. Do you mean, the root keys getting screwed up? That's akin to (and less likely than) a browser shipping a compromised CA store. https://www.iana.org/dnssec/tcrs these people can screw up the DNS PKI for the entire planet, I guess.

The roles between Web PKI and DNSSEC plainly do not line up for simple analogy statements to work. Again: what role in the DNSSEC PKI is getting compromised in your argument?

People say stuff like this all the time, as if "the blast radius" of fucking up .COM or .CO.UK or .IO was somehow small.

Constrained names aren't a solution for this problem. eMudhra had a thriving business selling certificates to US organizations. Unless you're really coming from a place of "we can't trust companies in India", the only coherent thing you can be arguing is that only one company should issue certificates from North America.

We tried that before. It was called Verisign. In a DANE-first world, it would again be Verisign. Immutably, and without accountability. There would be no transparency logs and no dis-trusting of them when they inevitably misissued at the behest of the United States Government.

The WebPKI's answer to this problem is to gradually ratchet requirements so that fewer and fewer companies can run CAs, and to build and coercively require transparency mechanisms like CT (something people on this thread inexplicably object to as "bullying") so those requirements can be enforced.

What on earth are you talking about? There is no concept of "issuing" anything in DNSsec/DANE. And why are you assuming DNS is frozen in its current state while the Web PKI can be ratcheted down? (And also, I'm not arguing for a "DANE-first" world. I'm arguing for a "DANE-and" world.)

What Verisign can do in all of this is change the NS and DS for a given domain, and Mallory can run different DNS servers for that domain then, giving out different data, including TLSA records. If they do this, they can also get ACME certs, except that will result in a permanent CT entry. Great feature! We should build a clone for DNSsec.

Will the owner of the domain notice either of these things? Hard to say. In both cases, you need to query something (CT logs or DNS caches) for your own domain, and check if things pop up you didn't put there. You can't query specific ISP's caches, that sucks. And it's only visible in DNS while under attack. Still, my gut feeling is more people will actually notice their NS getting messed with, because it happens by accident when they screw up their DNS. "credit card expired" and stuff like that. DS records are automatically monitored by any self-respecting DNSsec implementation for propagation & state triggering, but only for a given list of servers & caches. The attack could probably stay under the radar by keeping 1.1.1.1 and 8.8.8.8 intact, or at least some instances of it.

And then there's that tiny problem that only browsers really care about CT. Anything else is, at best, protected only if it can't be distinguished by fingerprint on TLS connect. DNS can only be subverted on a per-resolver/cache basis. Not really better, not worse, but different.

I have two takeaways from this. First, I should poke some people at IETF about inventing "DT", DNSsec Transparency. I expect it could probably be implemented in the DNS caches [in addition to end resolvers]. That'd be an interesting way of rolling things out: you can cover shitty client devices without changing them at all. If they use a DNS cache doing "DT", they'd be covered, even if not perfectly. The best time to do any of this would've been 2013, when CT came along. The second best time is now.

The other takeaway is, I should probably prod ENISA people to see if the EU bureaucracy can force some things. We've recently started working with them (mostly because the US apparatus is underfunded to hell and can't answer an e-mail with less than 6 months of delay), though we're only a tiny cog. And if the thought of the EU requiring some level of DNSsec or "DANE-and" gives somebody a heart attack: good!

And let's at least not lie about one thing: if "DANE-and" is being validated for a given target, that does prevent exploiting the eMudhra compromise.

P.S.: before we get in another pile of miscommunication: DNSsec/DANE validation goes on the end application or end host DNS component and the cache. The AD bit is a misdesign and scourge on DNSsec, should be ignored on any network packet, and is not what you build trust on. (AFAIK this is consensus in the DNS world, and one of the things getting ratcheted on their side.)

So you're upset that Google et al can't dictate who has authority over .com with DANE? Because yes, that's how DNS is supposed to work and also how domain validated web PKI works because that's just a glorified DNS consensus system which happens to give bullying power to browsers and root store operators. It would be up to ICANN (or countries for ccTLDs) to determine if the issue warants replacing the operator.

No, I think that state-controlled naming authorities shouldn't control cryptographic keys, for the reasons laid out on this and numerous other threads about DANE.

You know you're in trouble when you see people talking about "bullying" from the browsers, who are the only reason we have certificate transparency and don't have CAs literally selling root certificates to big companies to run their own private CAs.

CAs wouldn't be able to sell root certificates in the first place without the insane web PKI scheme where any CA can sign certificates for everything.

And yes, "bullying" is very much the appropriate word to use for the behavior of browser vendors towards the rest of the web ecosystem. For example it perfectly describes Google/Chrome's behavior in forcing Let's Encrypt to stop permitting non-web uses in their certificates, which didn't increase security for everyone but instead only caused problems for XMPP servers - temporary problems because those just started to ignore those fields so yay pointless work for everyone with not effect at all in the end.

At the point where you're trying to argue that it's a bad thing that Mozilla is able to force security standards on CAs at the point of a gun, our premises are too far apart to debate anything productively. Suffice it to say: losing that coercive threat is, to me, a big part of why you don't want DNS companies subsuming the role of CAs.

There is no one ecosystem. The world of web clients is highly diverse. Expecting every client implementation out there to converge onto one particular trust store is, first of all, magical thinking, but second of all, it reintroduces the single point of failure that web PKI is meant to avoid. The OS trust store is just one of many possible trust stores, there's nothing ordained about it. The problem is not with some standard, it's with a mistaken pre-conceived notion that there must only be one.

Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!

If you want all of the web clients on your system to use a centrally managed trust store, then it's your responsibility to (a) know and enforce what clients exist on your system and (b) configure them all to use that trust store.

I'm not making an argument for a single centrally-managed trust store. The thing that seems most obviously bad to me here, as a non-crypto-expert, is that any CA can issue a certificate for any domain. This is probably made worse because there are hundreds of CAs, and AFAICT functionally every trust store trusts all of the same ones.

I don't know how we fix this, but it does seem like a big problem.

> Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!

But this is the kind of DevEx decision to improve usability that goes against security, in the case of a shitstorm like this one. I mean, yes obviously everybody should be updating and refreshing all the systems running in all the environments daily/weekly but the reality in many (most) places is very different. So, it's an engineering decision with its trade-offs, as every decision. And OP was criticizing that decision(s) since this eMudhra issue happened now.

As as thought experiment: if no language were to distribute its own copy of root certs, we would have either docker images embedding them (which would be the same for dockerized workloads) or maybe some system would have been created to easily share a set of root certs within a cluster of applications.

Why don't these massive package managers use the host OS CA store, with an optional preconfigured "additional" store if clients need them?

Is the update cadence of CAs that different between the sources? Is the list of CAs included so different from OS vendors and the package managers?

Creating a dependency on OS specific APIs solves this one niche problem but causes a hundred more.

My understanding that go at least doesn't ship their own root store; they're shipping a patch to blacklist certain intermediate certificates instead, even if whatever OS-level root store would have accepted them.

Basically, the design didn't have a way to distrust anything other than removal from the root store. Which may still happen, but that will take longer.

> Also, it's quietly, in the email space, that we're seeing real innovation […]

Can you elaborate on this innovation in the email space around PKI?

Is it a out MTA-STS or something else?

DANE is a live thing in email. MTA-STS likewise uses DNS (unfortunately combined with a well-known HTTP URL path) to denote the equivalent of HSTS without everyone carting around a giant preload list. DKIM's signer-described partial-coverage signatures are pragmatic. DMARC's one-of-two alignment check is good and flexible. BIMI is good.

In general, email has used DNS to a greater extent than the web has to build trust mechanisms that don't depend on giant lists managed by a tiny number of browser vendors.

As the RFC says, MTA-STS exists in part to work around the fact that email providers generally don't do DANE.

That's an issue primarily with the hyperscalers, who're refusing DANE. Anything outside of Gmail, Outlook and the likes, you're much more likely to see DANE than MTA-STS.

(Most open source SMTP stacks don't integrate an HTTP client and are completely unable to do MTA-STS without hacks, patches or other fragile bolt-on bits.)

In other words, most Internet email is not sent using DANE.

Sure, but there are three major issues with your statement, all related to how you actually look at and count things.

First, not using DANE doesn't mean using MTA-STS. In particular, between the giant mail hosts, even if both sides publish MTA-STS information and validate MTA-STS elsewhere — I'm quite willing to bet they have a bunch of policy overrides for each other. In a way, MTA-STS is nothing other than standardizing those policy overrides so things don't break as much when they have to change something.

Second, in terms of installations, DANE absolutely wins, just because there are a few giant installations and a ton of small ones. Even if the quota on the small ones is bad, it'll still outweigh the giants by orders of magnitude. And it's technically far easier to roll out a new mechanism on a handful of giant, well-automated, well-monitored installations. They could all do DANE in a few weeks or months. They just don't want to operationally/politically. Conversely, updating millions of small shoestring installations to MTA-STS (or something else) is a herculean task.

(Related question: what about domains? That's yet another statistical perspective, considering a lot of domains point at Gmail & co.)

And lastly — I'm not sure your statement is in fact correct, because the medium-sized enterprise senders sending in bulk generally don't use the actual e-mail giant infrastructure (e.g. Outlook.com, Gmail.com), even if they're hosted at a corporate cousin (Azure, GCP). And that's probably enough volume to shift the outcome for "most Internet mail". For this, it'd be interesting to know what the various cloud-ish mail solutions (aaS & products) do, which I unfortunately don't.

This is like saying that DNSSEC is doing well in terms of "installations" because European zones tend to be signed by default by their registrars (which is security theater). What matters isn't the absolute number of zones signed; it's how widely used those zones are. Overwhelmingly, Internet email is sent without DANE consultation. By the numbers, MTA-STS (which is a marginal protocol) is wildly more important than DANE.

I agree, which is why I found it weird that you argued by way of "email providers generally don't do DANE."¹ - the wording as 'providers' makes an installation numbers based argument, or maybe legal entity numbers based. Definitely not mail volume. So I replied about installations, and then you switched to volume.

(¹ https://news.ycombinator.com/item?id=50173218)

I'm lost, but at least we agree that DANE is a nonfactor in email security, compared even to the (marginal) STS protocol.

Nah, they're both nonfactors probably. Hard to tell without real research.

(I meant I agree that counting by installations is odd.)

What's worse is, if this hadn't come to light, how would you even know when an attacker issues a cert for your domain?

Well you might think, oh I can monitor the certificate transparency logs.

I would say well, good luck downloading and parsing terabytes of certificates as they are logged. Then there are CAs whose logs will rate limit you in a way such that you can't catch up to the certs being logged with one server and IP. The system might "work" but not for domain owners. Good luck.

Reply on news.ycombinator.com