It's been a huge pain as we have encountered a ton of bugs and missing features in libraries and applications to reload certs like this. And we have some really ugly workarounds in place, because some applications place a "reload a consul client" on the same level of "reload all config, including opening new sockets, adjusting socket parameters, doing TCP connection handover" - all to rebuild a stateless client throwing a few parameters at a standard http client. But oh well.
But I refuse to back down. Reload your certs and your secrets. If we encounter a situation in which we have to mass-revoke and mass-reissue internal certs, it'll be easy for those who do. I don't have time for everyone else.
Do you promise to come back and tell us the story about when someone went on vacation and the certs issued on a Thursday didn't renew over the weekend and come Monday everything broke and no one could authenticate or get into the building?
I applaud you for sticking to your guns though.
And the conventional wisdom for application management and deployments is - if it's painful, do it more. Like this, applications in the container infrastructure are forced to get certificate deployment and reloading right on day 1.
And yes, some older application that were migrated to the infrastructure went ahead and loaded their credentials and certificates for other dependencies into their database or something like that and then ended up confused when this didn't work at all. Now it's fixed.
Except there are no APIs to rotate those. The infrastructure doesn't exist yet.
And refreshing those automatically does not validate ownership, unlike certificates where you can do a DNS check or an HTTP check.
Microsoft has some technology that next to these tokens they also have a per-machine certificate that is used to sign requests, and those certificates can't leave the machine.
Because we run on Azure / AKS, switching to federated credentials ("workload identities") with the app registrations made most of the pain go away because MS manages all the rotations (3 months) etc. If you're on managed AKS the OIDC issuer side is also automagic. And it's free. I think GCP offers something similar.
https://learn.microsoft.com/en-us/entra/workload-id/workload...
Individuals are in the same boat: if you're running your own custom services at your house, you've self-identified as being in the amazingly small fraction of the population with both the technical literacy and desire to do so. Either set up LetsEncrypt or run your own ACME service; the CAB is making clear here and in prior changes that they're not letting the 1% hold back the security bar for everybody else.
In the hackiest of setups, it's a few commands to generate a CA and issue a wildcard certificate for everything. Then a single line in the bootstrap script or documentation for new devices to trust the CA and you're done.
Going a few steps further, setting up something like Hashicorp Vault is not hard and regardless of org size; you need to do secret distribution somehow.
My dad still calls my terminals a "DOS window" and doesn't understand why I don't use GUIs like a normal person. He has his own business. He absolutely cannot just roll out a CA for secure comms with his local printer or whatever. He literally calls me to help with buying a PDF reader
Myself, I'm employed at a small business and we're all as tech savvy as it gets. It took me several days to set it up on secure hardware (smartcard, figuring out compatibility and broken documentation), making sure I understand what all the options do and that it's secure for years to come and whatnot, working out what the procedure for issuing should be, etc. Eventually got it done, handed it over to the higher-up who gets to issue certs, distribute the CA cert to everyone... it's never used. We have a wiki page with TLS and SSH fingerprints
This is fair. I assumed all small businesses would be tech startups, haha.
Paying experts (Ed: setting up internal infrastructure) is a perfectly viable option so the only real question is the amount of effort involved not if random people know how to do something.
Congrats for securing your job by selling the free internet and your soul.
If someone doesn’t want to learn then nobody needs to help them for free.
Perhaps spend some time outside your bubble? I’ve read many of your comments and you just do seem to be caught in your own little world. “Out of touch” is apt and you should probably reflect on that at length.
If we’re talking about businesses hosting services on some intranet and concerned about TLS, then yes, I assume it’s either a tech company or they have at least one competent engineer to host these things. Why else would the question be relevant?
> “Out of touch” is apt and you should probably reflect on that at length.
That’s a very weird personal comment based on a few comments on a website that’s inside a tech savvy bubble. Most people here work in IT, so I talk as if most people here work in IT. If you’re a mechanic at a garage or a lawyer at a law firm, I wouldn’t tell you rolling your own CA is easy and just a few commands.
Not to mention the massive undertaking that even just maintaining a multi-platform chromium fork is.
The nightmare only intensifies for small businesses that allow their users to bring their own devices (yes, yes, sacrilege but that is how small businesses operate).
Not everything is a massive enterprise with an army of IT support personnel.
Traefik (by default) will attempt certificate renewal 30 days before expiry. Perhaps the defaults will change if the lifetime becomes 45 days. I don't think it's possible to override this value, without adjusting the certificate expiry days, but I've never felt the need to adjust it.
For internal web services I could use just Let's Encrypt but I need to deploy the client certs anyways for network access and I might as well just use my internal cert for everything.
Or cheat and use tailscale to do the whole thing.
Having just implemented an internal CA, I can assure you, most corporations can’t just run an internal CA. Some struggle to update containers and tie their shoe laces.
I’ve done it in the past, and it was so painful, I just bit the bullet and started accessing everything under public hostnames so that I can get auto-issued Letsencrypt certificates.
https://www.digicert.com/blog/https-only-features-in-browser...
A static page that hosts documentation on an internal network does not need encryption.
The added overhead of certificate maintenance (and investigating when it does and will break) is simply not worth the added cost.
Of course the workaround most shops do nowadays is just hide the HTTP servers behind a load balancer doing SSL termination with a wildcard cert. An added layer of complexity (and now single point of failure) just to appease the WebPKI crybabies.
Just about every web server these days supports ACME -- some natively, some via scripts, and you can set up your own internal CA using something like step-ca that speaks ACME if you don't want your certs going out to the transparency log.
The last few companies I've worked at had no http behind the scenes -- everything, including service-to-service communications was handled via https. It's a hard requirement for just about everything financial, healthcare, and sensitive these days.
[proceeds to describe a bunch of new infrastructure and automation you need to setup and monitor]
So when ACME breaks - which it will, because it's not foolproof - the server securely hosting the cafeteria menus is now inaccessible, instead of being susceptible to interception or modification in transit. Because the guy that has owned your core switches is most concerned that everyone will be eating taco salad every day.
Of course, then there are the employees who could just intercept HTTP requests, and modify them to include a payload to root an employee's machine. There is so much software out there that can destroy trust in a network, and it's literally download and install, then point and click with no knowledge. Seems like there is a market for simple and cheap solutions for internal networks, for small business. I could see myself making quite a bit off it, which I did in the mid-2000's, but I can't stand doing sales any more in my life, and dealing with support is a whole issue on it's own even with an automated solution.
Someone that has seized control of your core network such that they were capable of modifying traffic, is not going to waste precious time or access modifying the flags of ls on your man page server. They will focus on more valuable things.
Just because something is possible in theory doesn't make it likely or worth the time invested.
You can put 8 locks on the door to your house but most people suffice with just one.
Someone could remove a piece of mail from your unlocked rural mailbox, modify it and put it back. Do you trust the mail carrier as much as the security of your internal network?
But it's not really a concern worth investing resources into for most.
Ah, the "both me and my attackers agree on what's important" fallacy.
What if they modify the man page response to include drive-by malware?
There’s nothing stopping Apple and Google from issuing themselves certificates every 10 minutes. I get no value for doing this. Building out or expanding my own PKI for my company or setting up the infrastructure to integrate with Digicert or whomever gets me zero security and business value, just cost and toil.
Revocation is most often an issue when CAs fuck up. So now we collectively need to pay to cover their rears.
The big question is what happens when (not "if") that happens. Companies have repeatedly shown that they are unable to rotate certs in time, to the point of even suing CAs to avoid revocation. They've been asked nicely to get their shit together, and it hasn't happened. Shortening cert lifetime to force automation is the inevitable next step.
You’re portraying people suing CAs to get injunctions to avoid outages as clueless or irresponsible. The fact is Digicert’s actions, dictated by this CA/Browser forum were draconian and over the top responses to a minor risk. This industry trade group is out of control.
End of the day, we’re just pushing risk around. Running a quality internal PKI is difficult.
Non browser things usually don’t care even if cert is expired or trusted.
So I expect people still to use WebPKI for internal sites.
Why would browsers "most likely" enforce this change for internal CAs as well?
That said, it would be really nice if they supported DANE so that websites do not need CAs.
So yes, you had to first ignore the invalid self-signed certificate while using HTTPS with a client tool that really, really didn't want to ignore the validity issue, then upload a valid certificate, restart the service... which would terminate the HTTPS connection with an error breaking your script in a different not-fun way, and then reconnect... at some unspecified time later to continue the configuration.
Fun times...
Isn't this a good default? No network access, no need for a public certificate, no need for a certificate that might be mistakenly trusted by a public (non-malicious) device, no need for a public log for the issued certificate.
'ipa-client-install' for those so motivated. Certificates are literally one among many things part of your domain services.
If you're at the scale past what IPA/your domain can manage, well, c'est la vie.
I think folks are being facetious wanting more for 'free'. The solutions have been available for literal decades, I was deliberate in my choice.
Not the average, certainly the majority where I've worked. There are at least two well-known Clouds that enroll their hypervisors to a domain. I'll let you guess which.
My point is, the difficulty is chosen... and 'No choice is a choice'. I don't care which, that's not my concern. The domain is one of those external things you can choose. Not just some VC toy. I won't stop you.
The devices are already managed; you've deployed them to your fleet.
No need to be so generous to their feigned incompetence. Want an internal CA? Managing that's the price. Good news: they buy!
Don't complain to me about 'your' choices. Self-selected problem if I've heard one.
Aside from all of this, if your org is being hung up on enrollment... I'm not sure you're ready for key management. Or the other work being a CA actually requires.
Yes, it's more work. Such is life and adding requirements. Trends - again, for decades - show organizations are generally able to manage with something.
Literal Clouds do this, why can't 'you'?
You're managing your machine deployments with something so of course you just use that that to include your cert which isn't particularly hard but there's a long-tail of annoying work when dealing with containers and vms you aren't building yourself like k8s node pools. It can be done but it's usually less effort to just get public certs for everything.
To your point, people don't, but it's a perfectly viable path.
Containers/kubernetes, that's pipeline city, baby!
I am not willing to give credentials to alter my dns to a program. A security issue there would be too much risk.
Introduce them to the idea of having something like Caddy sit in front of apps solely for the purpose of doing TLS termination... Caddy et al can update the certs automatically.
I haven’t used BIG IP in a long while, so take this with a grain of salt, but it seems to me that it might not be impossible to get something going – despite the fact that BIG IP itself doesn’t have native support for ACME.
Two pointers that might be of interest:
https://community.f5.com/discussions/technicalforum/upload-l...
https://clouddocs.f5.com/api/icontrol-rest/APIRef_tm_sys_cry...
Those tend to be quite brittle in reality. What’s the old adage about engineering vs architecture again?
Something like this I think: https://www.reddit.com/r/PeterExplainsTheJoke/comments/16141...
For some companies, it might be worth it to throw away a $100000 device and buy something better. For others it might not be worth it.
Giving the TLS endpoint itself the authority to manage certificates kind of weakens the usefulness of rotating certificates in the first place. You probably don't let your external facing authoritative DNS servers near zone key material, so there's no reason to let the external load balancers rotate certificates.
Where I have used F5 there was never any problem letting the backend configuration system do the rotation and upload of certificates together with every other piece of configuration that is needed for day to day operations.
Don't expect firmware / software updates to enable ACME-type functionality for tons of gear. At best it'll be treated as an excuse by vendors to make Customers forklift and replace otherwise working gear.
Corporate hardware lifecycles are longer than the proposed timeline for these changes. This feels like an ill thought-out initiative by bureaucrats working in companies who build their own infrastructure (in their white towers). Meanwhile, we plebs who work in less-than-Fortune 500 companies stuck with off-the-shelf solutions will be forced to suffer.
Or request a certificate over the public internet, for an internal service. Your hostname must be exposed to the web and will be publicly visible in transparency reports.
Key loss on one of those is like a takeover of an entire chunk of hostnames. Really opens you up.
That doesn't seem like the end of the world. It means you shouldn't have `secret-plans-for-world-takeover.example.com`, but it's already the case that secret projects should use opaque codenames. Most internal domain names would not actually leak any information of value.
The closest thing is maybe described (but not shown) in these posts: https://blog.daknob.net/workload-mtls-with-acme/ https://blog.daknob.net/acme-end-user-client-certificates/
(disclamer: i'm a founder at anchor.dev)
> The CSR relayed through Anchor does not contain secret information. Anchor never sees the private key material for your certificates.
Give us a big global *.local cert we can all cheat with, so I don't have to blast my credentials in the clear when I log into my router's admin page.
For the other case perhaps renew the cert at a host allowed to do outside queries for the dns challenge and find some acceptable automated way to propagate an updated cert to the host that isn't allowed outside queries.
There are lots of systems that allow you to set rules for what is required to merge a PR, so if you want "the tests pass, it's a TXT record, the author is whitelisted to change that record" or something, it's very achievable
If it's just because your DNS is at a provider, you should be aware that it's possible to self-host DNS.
Someone will fuck up accidentally, so production zones are usually gated somehow, sometimes with humans instead of pure automata.
Giving write access does not mean giving unrestricted write access
Also, another way (which I built in a previous compagny) is to create a simple certificate provider (API or whatever), integrated with whatever internal authentication scheme you are using, and are able to sign csr for you. A LE proxy, as you might call it
If you are not in a good position in the internal organization to control DNS, you probably shouldn't handle certificate issuance either. It makes sense to have a specific part of the organization responsible.
You can do nothing except twiddle your thumbs while it times out and that may take a couple of days.