Let's Encrypt AUTHZ Reuse and Eternal Account Key | Hacker News Reader

Let's Encrypt AUTHZ Reuse and Eternal Account Key

dan.enigmabridge.com

107 pointsdc35239 comments

I think it's also important to note... you don't need the account key at all if you can re-do the domain verification every time.

In most cases I delete the account key right after verifying the domain and getting a cert, and let certbot generate a new one every time.

We route the '/.well-known/acme-challenge' request to the standalone server (we use haproxy ACLs 'path_beg' to route it to a dedicated port on localhost) so it hasn't been a problem running the validation every time for every new cert.

And per TFA, the timeout for validation on their side (what they call Authz or authorization token) is about the same as the timeout for renewal, so you can't really depend on not needing to validate the domain even with the account key. I'd much rather have the code path be identical every time, rather than different for new cert vs renewal cert, and then also different every other time you renew due to authz expiry.

> In most cases I delete the account key right after verifying the domain and getting a cert, and let certbot generate a new one every time.

An alternative would be deactivating the authorizations.

In general, you'll automate lets-encrypt. That means you'll have some file or group of files (a cronjob, systemd timer, or what have you) that's capable of producing new keys.

If an attacker can read files on the filesystem as root, then they can simply read whatever automation scripts you have and issue new certs as well.

This isn't always true (e.g. if it depends on an IAM role, they'll have to get data from the instance metadata service as well), but the point is generally true.

If an attacker has root filesystem access on a box which can issue certs, it makes miniscule difference whether they need to grab an account key or bash script; either way you've probably lost.

You could also, however, have more locked-down boxes which issue certs, encrypt them for their intended hosts, and transport them over, at which point all of your host boxes no longer have account keys to worry about.

You could also encrypt-and-ship the account key from the box to elsewhere, and then remove it.

Frankly, you're making a lot of fuss about absolutely nothing.

The difference is that I need to access your file system only once to get your account key. I need permanent access to exploit your automation.

Attackers can also scale revenues, potentially, by selling account keys to third parties.

For most types of automation they only need to access your filesystem once as well, but accessing the filesystem as root once also implies they can again (albeit at increased risk of detection).

Your second sentence is pure FUD imo.

If you have to authenticate the domain every time, there's a good chance you capture that specific HTTP request traffic and log it to another server which isn't compromised. You now have an audit trail of the cert being issued and you have a chance of catching it.

If the account_key "walks away" and within 90 days they can get a cert without anything hitting your server, you have no audit log of the certificate being issued and are less likely to know your TLS is owned.

So if I understand correctly, the account_key is an unnecessary risk with very little return. (see my other post). I just immediately delete it every time, I don't see the point of keeping it.

Even key revocation does not require the account key, just the certificate private key. So really, what is the point? Unless there's something you actually need the account_key for... I feel like this was a failure to KISS.

EDIT: Mentioned downthread, there are also Certificate Transparency logs which can help you catch an unauthorized issuance, but that's not going to let you wire up an instant SMS through Twilio on every domain validation.

With the stolen account key scenario, you still get an audit trail through Certificate Transparency, as Let's Encrypt logs all certificates they issue.

> Even key revocation does not require the account key, just the certificate private key. So really, what is the point? Unless there's something you actually need the account_key for... I feel like this was a failure to KISS.

Can you revoke with either one? If someone steals my account key, and creates a new certificate private key do I have no way of revoking that since I don't have the new certificate private key?

Yes, both keys can be used to revoke, so as long as you still have either the account key or all certificate private keys, you're good.

> The surprising aspect is that Authz has a validity of 300 days (which is likely to be increased)

This is not true. Authorization lifetimes are only going to get shorter and shorter [1], and have already been reduced to 90 days.

[1] https://community.letsencrypt.org/t/upcoming-api-changes/179...

I've updated the blog post. I don't think, though, it has any material impact on the rest of the text.

Why not? Right now, compromised account keys give you the ability to issue certificates that are valid about as long as certificates are valid (or, in the worst case, +90 days from the original expiration date). Just about any scenario where the account key is compromised is one where the certificate's private key is compromised as well (not to mention typically one where you can just solve another challenge), so you're not really in a much worse position. Even less of a concern once authorizations are valid for only ~7 days.

> I've updated the blog post

You still state in your updated post that: "The surprising aspect is that Authz has a validity of 300 days (which is likely to be increased)" [emphasis mine]. This would appear to be incorrect based on the source cited above , where it is stated at "Eventually, we'd like to make authorization objects much shorter than certificate lifetimes, probably 7 days."

Perhaps it's worth updating the post again in light of this?

It is an interesting one - here is the original source (and there were many more mentioning 300 days).

https://community.letsencrypt.org/t/dns-authorization-lifeti...

It is from 14 June 2016 (a few months back), @pfg states "The CA/B Forum is currently developing new rules for domain validation and is probably going to settle on a validation period that is significantly longer than the 300 days currently in use ..."

Is there an authority to say which way it will go?

I think it's 36 months right now, but that's a Baseline Requirements thing, Let's Encrypt can (and does) use shorter periods. 36 months is the upper limit for all CAs. Let's Encrypt currently uses 90 days, and will go to something like 7 days in the near-future[1].

[1]: https://community.letsencrypt.org/t/upcoming-api-changes/179...

If you don't mind, I'd like to add to your blog post.

Token issues are something we solved about a decade ago. As to why LetsEncrypt has yet to learn these lessons, I leave that as an exercise to the readers and those familiar with security issues to discover for themselves.

Can I attribute this to a particular name?

The authorization lifetime should only last a few hours if that at best.

Let's Encrypt is slowly learning the lessons one should have learned a decade ago regarding tokens.

That does not give me hope for their future.

This is not a traditional token use-case (as you would use for e.g. a password reset email). We're talking about long-term identity keys (the account key) which are associated with authorizations for FQDNs, giving the owner of that key the ability to issue certificates for those FQDNs. The risk is not related to any kind of token, but rather to losing the long-term identity key, which really is no different from losing credentials for a more traditional CA (who can re-use these authorizations for up to 36 months).

The risk is that you're not using instantly-terminating tokens.

I've had over 20 years (way longer if you count BBS systems which had the exact same problems then as we have now) to play with this. I retired on this very concept.

How do authenticate yourself to get a new instantly-terminating token?

> I did expect that each request includes a fresh verification of the domain control. And I also expect that substantial majority of people out there believe in this process / data flow as that’s how certificates have always been verified – for each new certificate, you either have to send an email from a given domain or put a file to a web server’s folder.

This is not correct. Certificate Authorities are allowed to reuse authorizations for (I believe) up to 36 months, so Let's Encrypt is already significantly below that (with 90 days right now, and probably closer to something like ~7 days soon). Many CAs I've used in the past worked like this.

This is interesting in the context of buying and selling domain names. Let's say you pay big $$$ for a fancy .com because you're launching a new product with that name or whatever. The seller could actually possess certificates (or authorizations) that allow them to MITM your fancy expensive domain for up to, what, 3 years. Never really thought of that.

CT logs could prove on the date of purchase that your new domain no longer has any valid certificates in the hands of third parties, but if the third party still holds an authorization, they could sneakily obtain new certificates long after the sale? Or - are authorizations also logged somehow to CT-like logs?

Yeah, it's definitely something I'm hoping the CA/B Forum will address at some point. Something like a month seems like a sweet spot between not being annoying for new customers when they're in the process of getting multiple certs and not being a huge deal when domains are transferred.

I'm not aware of any kind of "Authorization Transparency" effort. Let's Encrypt were thinking about a way to share logs at some point (IIRC), so it might get included in those, but that's a maybe for the future and not a general solution for all CAs.

>And I also expect that substantial majority of people out there believe in this process / data flow as that’s how certificates have always been verified – for each new certificate, you either have to send an email from a given domain or put a file to a web server’s folder.

Err... what? I don't think any CA validation process has ever involved sending e. mail, and I'd be very concerned if it did, since the origin of an e. mail is not in general authenticated. Unless it's in response to a CA e. mail which includes a token, of course.

'Obtain reusable authorization' followed by 'issue certificate' is hardly unprecedented, and it's a sensible way to structure things if you don't have a commercial interest in applying the friction of a financial transaction to the act of issuance. Startcom, for example, used this model (there are many issues with them, this wasn't one of them). I think CAcert may also use this model, unrecognised though they may be.

The ACME protocol is also a standard. It's right there, I'm surprised people aren't reading it, especially when they're rigging up custom solutions. https://tools.ietf.org/html/draft-ietf-acme-acme-02

If you're upset about the expected operation of a service based on open-source software (github.com/letsencrypt/boulder), then I'd strongly recommend reading the code before use. Additionally:

"I also expect that our Enigma Bridge Cloud HSM service (check our PKI product with hardware key security) can securely control storage and use of the account key to prevent thefts of the keys and their subsequent misuse."

Using this as a way to promote your product and not reading the code first is a serious failure of tradecraft on your part.

Or just destroy the auth key once a certificate is issued. No need to keep it around. Can go through the expected validation process for a new one when renewing.

Absolutely not. You need it for revocation. Destroying it is thus dangerous and reduces your security.

Edit: I'm wrong, see pfg's comment below. See my top-level comment for why I don't think it matters still.

Revocation is also possible using the certificate's private key, but I agree that the risk of keeping the account key is not worth the hassle of deleting it straight away. As it stands, authorizations are valid as long as certificates, so the worst case is you're compromised on the 89th day of validity and get another certificate valid for the next 90 days. Soon, that'll be ~7 days, so even less of a concern.

So do you imply that the account key is created from scratch for every new certificate?

Why we were surprised (and we don't say the implementation is necessarily wrong!) is that I can use the account key anywhere. If the genuine user keeps refreshing authz's, it will keep the stolen account key operational as well.

That's my understanding. I may be wrong, but if so, I don't quite yet understand the logic behind authz.

No, account keys are typically persisted and re-used, though specifics depend on the client implementation. I still think this is largely a non-issue. If you compromise the target system, you can just keep stealing private keys as they are being renewed. This would even decrease the chances of being detected, because using the stolen account key to issue new certificates would lead to those certificates popping up on various Certificate Transparency log servers, which might be noticed by the domain owner. You're already in a game-over scenario if you don't detect the compromise, the account key doesn't make things significantly worse.

Think of it like the credentials for a hosting provider where you bought your traditional SSL certificate being stolen. Not really a new threat.

If you are creating a new account key with a new authz, but the old account key still gives you access to renewal certs, that IMO is absolutely a bug.

The authz should be tied to the { domain, account_key } tuple, not just the { domain }.

Can anyone confirm?

That's not a bug.

I have multiple servers that serve { domain } in dns round-robin. Each of them has a unique account key, but they also each issue certs for the same domain.

This is a totally normal and valid use-case, and allowing only a single account key for a domain at a time would ruin it.

This is contrary to their documentation at [1].

  I'm sure this is what was meant, but just to confirm and spell it out, the authorization
  has to belong to the subscriber account that's making the request right?

  @tialaramex Apologies that it wasn't clear - you are absolutely correct. Authorizations
  are only reused within a subscriber account.

  tialaramex: If account A asks to create an authz for example.com, succeeds in validating
  it, and then account B comes along and wants an authz for example.com too, that should
  result in a fresh authz regardless of this setting.

  Correct. Account B's request would result in a fresh authz in pending state.

It's fine for multiple different servers with different account_keys to be able to each independently validate a domain and get their own unique authz tokens, but one server with account_key A must not be able to piggyback off of another server's authz done under account_key B.

Since, remember, it is trivial for anyone to get a new account_key, there is absolutely no validation that occurs to setup a new account key for a given domain. This is why I don't understand the purpose behind the account_key in the first place.

[1] - https://community.letsencrypt.org/t/upcoming-change-valid-au...

"Soon, that'll be ~7 days, so even less of a concern."

If you aren't authorizing and revoking immediately after delivering data, you're doing it wrong and opening yourself up to attack.

Source: Four penny auction websites using the same code were compromised by me (at their request - hired hacker.) I took their entire auction winnings after they gave me authorization to test their systems. It took me less than 20 minutes.

I walked away with $300,000 because in my contract I told them I keep whatever I compromise and steal. They agreed.

If you can't handle rapid token usage/generation, don't even bother using tokens. Token revocation is one of the harder things out there to get done properly. You're better off with a challenge/auth EVERY SINGLE REQUEST at the expense of page loading times.

Again, this isn't about tokens. We're talking about a private account key (like, an actual RSA or ECC key), which is associated with FQDNs for which the holder of an account key has the ability to issue certificates.

I'll try to phrase this as an example: Let's say you run an online store for music downloads. One implementation might be to send the user an email with a download link, identified by a token. In this case, it would make sense for this token to expire after the user successfully downloaded the file, and doing otherwise would probably be bad for your business. On the other hand, if your online store involves a login process you need to complete before you can actually download the file (the file being the domain authorization in this example), that's fine and you might not want to limit the user to one download.

My recommendation is to use the DNS challenge implementation with an appropriate DNS provider (I use Route 53): you can do all your certificate generation on a single locked-down machine and then just distribute the certs out to your machines. No need to deal with path_beg or check which host in the fleet is responding to the acme-challenge request.

"This is not true any more with re-usable AUTHZ tokens"

I must re-iterate my own 10+ year experiences with Token auth - once authorized, end that token, period, or you're asking for problems.

Feel free to downvote - I retired because I was right and proved it AND GOT PAID FOR IT. Come retire at 28 years of age, guys, if you bother to listen.

Reply on news.ycombinator.com