Automatic SSL Certificates for internal IP's for home k8 setup using LetsEncrypt
techprowd.com
techprowd.com
I recommend this not just for internal IP setups, for actually for all setups, since DNS verification is more robust than HTTP verification, particularly if you have issues with load balancers, or if Let's Encrypt decides to deprecate a protocol again [1].
[0] https://cert-manager.io/docs/configuration/acme/dns01/#suppo... [1] https://community.letsencrypt.org/t/upcoming-tls-sni-depreca...
The two likely reasons for such deprecation would apply just as well:
1. Updated Baseline Requirements or a programme policy requirement at any of the major root trust stores could forbid this challenge or require it to be substantially modified, obsoleting it in its current form.
2. The BRs don't change but Let's Encrypt finds they need to adjust this particular implementation in a non-compatible way so they deprecate the current challenge.
It can be easier to do DNS challenges, but it can also be very rough, depending on all the moving parts in your system.
However, in practice, the DNS challenge (which demonstrates control over DNS) is greatly preferred over HTTP/TLS challenges (which demonstrate control over a single port).
DNS is likely to be the only way to get a wildcard certificate, and HTTP/TLS will likely end up further restricted once SRVNames in certificates can be gracefully rolled out.
As such, deploying the DNS based control is absolutely the best thing to do, and HTTP/TLS should be seen as legacy-compat fallbacks that may become more difficult in time. Either certificates become less scoped than “entire domain” (as they are today with dNSName SANs) or it becomes more difficult to use a single port to prove authorization for an entire domain.
Any HTTP/ALPN request first begins with DNS, so if you’re trying to compare those, they all share the same base issue. In theory, this can be mitigated by DNSSEC, but that’s not relevant when comparing these validation methods.
However, both the HTTP and ALPN methods only demonstrate control over a single port (or .well-known resource), while the DNS method demonstrates the full ability to alter any/all names.
You're not wrong, but this assumes that you use the your 'service hostname' for verification as well, rather than using CNAMEs.
So let us say you want to have "svc1.example.com" in your cert: you could put the ACME challenge under there, but if you have anycast delays that's a problem (as you mention). (A kludge is putting a 'sleep' somewhere to allow for propagation.)
So instead what you can do is have "_acme-challenge.svc1.example.com" be a CNAME that points to (say) "_acme-challenge.svc1.dnsauth.example.com". This sub-domain is not anycast, and may actually be a single machine that is used solely for this purpose.
The LE/ACME server goes to your main domain, finds a CNAME, and follows that to the real record and verification is achieved:
* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/
* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
The CNAME has to be set up initially, but can be left lying around otherwise.
This is how $WORK deals with getting LE certs for internal domains: we create a CNAME record (but no A records) for the internal hostname in our external DNS that point to our "dnsauth" domain which gets updating by internal clients via an API.
Do you use a custom acme/dns updater for automatic renewals?
[ed: ie - if I understand correctly, I could point: _acme-challenge.example.com via CNAME to auth.other.example.net - but then I'd like a command to check renew my example.com certs - and it would ideally use an api/dns update to manipulate the auth.other.example.net TXT (or CNAME to something like a15ce5b2-f170-4c91-97bf-09a5764a88f6.auth.acme-dns.io) record when I ask for a check/update of '*.example.com' certificate.
As far as I'm aware, most tooling assumes that you can/will (programmatically or manually) update the _acme-challenge.example.com record directly when issuing/updating an example.com certificate?]
First we use standard LE/ACME clients: either certbot or dehydrated. They ask for something like svc1.int.example.com ($DOMAIN).
In the hook script(s)† we manipulate the $DOMAIN string to put it into dnsauth.example.com ($AUTH_ZONE) sub-domain and send that new string to the DNS server that handles the dnsauth zone (and only that).
Before all of this we would have set up, in our public-external DNS, a CNAME record to point svc1.int to svc1.int.dnsauth.
The ACME client only thinks about $DOMAIN and the cert-issuing LE server only thinks about $DOMAIN. But the "in between" does not: by doing text manipulation (expr(1) is handy in shell scripts), and DNS redirects, the "in between" uses not-$DOMAIN for verification, but rather TXT records in $AUTH_ZONE.
All with standard ACME clients and some jiggery pokery.
We ended up creating some custom scripts called via SSH, but there are (now) DNS servers written specifically to handle REST API calls [0] and one can use lexicon [1] for just about any commercial DNS service.
[0] https://github.com/joohoi/acme-dns
[1] https://github.com/AnalogJ/lexicon
† dehydrated has deploy_challenge() and clean_challenge() functions in its example hook script. I'm sure most ACME clients have something similar.
In my case I'm mostly interested in delegating a domain/sub-domain somewhere I can easily update (be that run my own dns, host it somewhere with an api) - while having my main domains on a more boring/static dns infrastructure - yet still easily get certs for things like imap.example.com - which would not run a web server. And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.
Well, depending on the OS, you could start up a web server during the LE verification process and then bring it down once that's done. You'd only have to run in on port 80 for probably less than a minute.
But yes, you could this mechanism to have "_acme-challenge.imap.example.com" (which is what the ACME protocol uses) be a CNAME to point to something.auth.example.com that is more dynamic. Or even a completely different domain like foo.bar.example.ORG.
In your example.com zone file you'd put NS and A(AAA) records to point to the DNS server that handles the queries for the auth sub-domain.
> And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.
It's easier to run the ACME client on the host in question, and I'm not sure what it gains you to have it run somewhere else. That being said, there are ACME clients with a bit of a focus on being run 'remotely' from where the certs actually live:
* https://github.com/srvrco/getssl
This is probably for shared-hosting scenarios where cron is not accessible.
IMHO though, if you have access to the CLI on the host running the TLS service, it's best to run things there.
In general I'd prefer the certs be something the services get via configuration mgmnt - while the cert service can run via cron and make sure certs are valid an present.
In particular, I don't want my smtpd server to have write access to my dns, if I can help it.
CAs are required to run full recursive resolvers, up to the root, and can’t just point at someone else’s DNS infrastructure. Which, if you think about it, is what you want: you don’t want the CA just trusting someone else is being honest, you want to go to the authoritative source.
having multiple nameservers is pretty standard and often mandatory requirement set by the NICs. Its just that my secondary is sometimes not fast enough to transfer the zone after a change notification which triggers this issue.
Also, retrying after an authoritative nameserver said there is defacto no record seems pretty wrong to me... i doubt they use google DNS or anything really. in order to avoid caching issues they very likely resolve names recursively without (the usual) caching
As time goes on the value of having services running as appliances is becoming more and more valuable.
Whether this is worthwhile or not is debatable. Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use?
The other option for internal certificates is to get a wildcard of *.internal.mydomain.com, and spread that wildcard certificate around your network.
The final solution is run your own certificate authority and trust it on every browser. For some reason when you import a root certificate you can't typically allow that CA to only be used to authenticate a given subdomain. There are x509 constraints you can use in setting up the CA, but that's rare too, and I'm not sure every tool uses it.
In any case, if you go for an internal DNS provision, make sure you set use-application-dns.net to NXDOMAIN on your internal dns server to override DoH too
You're probably aware of tools such as dnsdumpster (dnsdumpster.com) that attempt to map internal networks using dns. It is common to use descriptive prefixes (eg. intranet, gw, fw, dns, jira, mysqldb, etc) that also give some insight on the internal tools and topology. Using the CT logs you can also collect over time changes in network, for servers/endpoints with non-wildcard certificates.
Its not a huge risk. But why give a map of the place to a potential attacker, when - as you quite well proposed - you can use simple ways of avoiding it?
If you have a per-host DNS based with no valid A records, DOH could well mean worse performance in future as devices don't listen to canary domain and only query locally once getting an NXDOMAIN (and even then might not). You're still leaking the CN and SAN entries via CT.
If you run your own CA, every device has to have it installed, and it's a major security risk as losing control of the CA means all your machines can be compromised.
Between certificate transparency and DNS over HTTPS, there's no perfect answer.
While, true, there are some workarounds, that are, ehm, workaroundy.
1. For _your_ domain, you can have CAA records in your DNS. (That solves it for your domains, not for others, if they do not use CAA)
2. Some CA implementations, like FreeIPA one, allow you to issue certificates only for your own domain. They will refuse to sign a certificate for any other domain.
So while it is not 100% solution, it is 80% one.
> In any case, if you go for an internal DNS provision, make sure you set use-application-dns.net to NXDOMAIN on your internal dns server to override DoH too
Also make sure that the browser honors this. Firefox honors canary domain only when in 'auto DoH' mode, but not, when the user has explicitly configured DoH. From https://support.mozilla.org/en-US/kb/canary-domain-use-appli...:
> The canary domain only applies to users who have DoH enabled as the default option. It does not apply for users who have made the choice to turn on DoH by themselves.
This is especially problematic if I want someone else to trust my certs in their browser, I'm asking them to trust any certificate I generate, including google.com, mybank.com, etc.
For some other purposes, where the Web PKI isn't applicable, you might have a solid argument for why people should trust your PKI. But then they aren't trusting certificates in their browser but in some other likely very constrained system that's easier to reason about.
For example I've operated (for past employers) a service that was available over HTTPS to a handful of insurance and financial outfits using mutual TLS. I'd issue client certificates for them, and then they'd use their private key and their certificate to access the service over HTTPS. In that case I ran the CA, and so they're trusting me. But only narrowly to identify customers of the service I run, so it's not tricky to reason about.
Now, technically you could use certificates from Let's Encrypt as client certificates, they have the right EKU for that purpose but then your identity has to be an Internet FQDN and you need new certificates every ninety days, which sucks. So although I'd have accommodated a paying customer who firmly insisted on doing that, none of them did.
I can secure my servers with my own CA, again that's fine - I trust myself.
If you want to verify my servers signed from my CA though you would need to trust my CA.
Currently that means you would need to import my CA's root certificate. That's bad, it's also the way many corporations work, I've had third party suppliers try to get me to trust their own internally generated root certificates too.
A better solution would be for the user to be able to import a root certificate for use for a set purpose. I'm happy to take foobar inc's certificate to authenticate *.foobar.inc, but not to trust anything else.
For some reason browsers don't allow that - it's all or nothing.
Absolutely. Name constraints. It's in the spec, but no one seems to implement it. This would fully make PKI useful. Typing x509 certificate (validation) to a delegated domain would allow people to import Root CAs from a variety of places while compartmentalizing security.
Right now, if you trust a Root CA cert, you trust any domain it signs. This has been a glaring problem for years. I never thought we'd have cheap certs for everything and LE came along and blew my mind.
Pretty much this. What does it matter if you know certain hostnames or internal IPs on my network? It's all firewalled anyway, and if it wasn't it would be trivial to find them out on your own...
This is just security by obscurity. It doesn't add much if you already implement actual security in the form of firewalls, network segregation, etc.
"Hi it's Bob down in IT, any chance you could reboot 'funkyserver123', it's on 172.18.12.5, for some reason I can't get in"
Edit: I don't know why this is being downvoted. The OP is about a home k8s setup, so unless you live in some castle or palace I very much doubt there will be a Bob down in IT.
Your options at that point are a central 'well known' directory that different hosts can write to (I recommend sshfs), different directories on one host that are checked for any valid file in any of them (by default) or by hostname match in specific, etc. The details depend on your security model.
So iiuc there is no split horizon, it’s just that the sites would only work for the author.
ISRG is required to keep enough information about the issuances they make to allow them to usefully diagnose problems after the fact. Ideally when we discover a problem it will be possible for the issuer to go back and figure out which (if any) previously issued certificates were affected so that these certificates can be revoked if appropriate.
But although they had at one point planned to publish more of this information, they do not in fact do this routinely.
for IP I dont think lets encrypt logging the ip address publicly (let me know if I wrong about it), since I use dns-01 I can generate SSL from anywhere.
Yes, if you override DNS (or just provide it) internally then the IP is hidden. The hostname is still leaked though - for example Geodynamics limited have server names of the format fvdcsev01 and fvdccit03 -- https://crt.sh/?id=2381703940
And yes be aware that through this, it works fine but you are also exposing your internal infrastructure details through dns.
I'm not seeing a big issue, just be aware of it.
Sure, that's technically correct but a wee bit overly pedantic. When technical people speak about SSL/TLS certificates it's common parlance to say "SSL" and everyone usually knows what you're talking about, which includes TLS, and whatever other new acronym might come down the pipe in the future.
My most used skill is to make sure i'm pedantic aka 'so we need to calculate this from that after this? and we need accuracy of 0.32? And what should that button do exactly?'
It is not SSL its TLS and i don't expect everyone to get it but its still wrong.
The weirdes thing in IT is, that i don't know any other word which is so missused then SSL.
I have yet to apt-get install opentls on any webserver.
I worry that I'll forever have to use both terms, because while SSL is the term that communicates better to semi-literate audiences, I worry that some security expert will assume I don't know anything because I'm using the "wrong" term.
My perspective was (and is) that for portable devices (phones/laptops) that are interfacing with locally hosted services, having SSL for those services is really important because your device probably isn't configured to check what network it's on before automatically pinging 192.168.1.x. This is doubly important if you have other people occasionally hopping onto your network and connecting to those same services. It's imo bad practice to ask everyone connecting to your network to install certificates or set up a certificate manager. I wouldn't do that for any of my personal devices if someone asked me to.
To push this a step farther, I imagined a world where my services could handle not just renewing their own certificates, but also updating their addresses if they were moved to a different network/address. If I build a physical device to give to someone, I'd like them to be able to plug it into their network, go to a web URL, and have everything just work -- no messing around with their internal DNS settings or worrying about whether they're using DNS over HTTPS in Firefox.
[0]: https://danshumway.com/blog/encrypting-internal-networks/
I’ve heard guix has some kind of container management thing. I’ve been thinking about trying it anyway.
MetalLB for your loadbalancing needs, prometheus/grafana etc.
And microk8s supports multiply nodes now as well.
I've been using it couple of years for many types of workloads and it's been a pleasant experience.
Ansible + docker-compose.yml to manage apps on 1-3 servers. You just install docker and that's it. Swarm configuration is 95% similar to compose file you use locally if you're into developing with Docker.
It has couple of edges if you try to use it for complex setups but for apps with < 5 devs and < 10 services it's really simple. You can still migrate to K8s when you hire dedicated DevOps team because you're running containers and all your setup documented in docker-compose file.
it's understandably complex, even if many parts are pretty standard (eg. the sysctl stuff, and installing dependencies is basically dnf/yum/apt/apk or exit and let the user do it).
since the most error prone parts were/are setting up the TLS stuff that got automated first (in the form of kubeadm install), and the rest just remains in "run this" form.
but the k3s installer is just a one liner call to a bash script. though then you have to make sure to include the magic env vars to get what you want.
I find it incredibly annoying that i can't tell chrome to use THIS CA Root Cert only for *.mydomain.com and not my banking domains, email, etc.
My public services all run out of .mydomain.com and all my internal services run out of .internal.mydomain.com
I have my internal dns set to resolve any .internal calls to an internal load balancer which hosts the ca certs.
The downside is that all internal services are ssl terminated at the load balancer, but this makes handling internal certs easy as they're rotated in a single location. This is Good Enough for my homelab.
[0] https://github.com/inlets/inlets-operator
Feel free to check it out in this tutorial: https://docs.inlets.dev/#/get-started/quickstart-ingresscont...
https://github.com/joohoi/acme-dns/
https://github.com/joohoi/acme-dns-certbot
A simplified DNS server with a RESTful HTTP API to provide a simple way to automate ACME DNS challenges.
"NoScript detected a potential Cross-Site Scripting attack from https://www.techprowd.com to https://carbon.now.sh"
Images are failing to load too. Not sure what's going on.
edit: Probably some misusing of DNS rather than actual attack but who knows. Author should fix the site.
I bought a real domain name, told my UBNT USG that was the domain for my network, set up the dns servers to use digital ocean, used jetstack's cert-manager [0] to acquire the a wildcart cert using DNS01 instead of HTTP01, and use kubed [1] to synchronize the TLS cert across namespaces. One key thing to consider is that you really should ensure that you use the staging let's encrypt server to test out issuance and see your browser complain about warnings before you switch to production let's encrypt.
Honestly, I don't mind that the cert requests for my domain show up in a CT log.
https://github.com/FiloSottile/mkcert
to automate setting up local CA and making it trusted.
The fundamental problem is that this CA that you generated gets basically the same trust level as a public CA, but it's just sitting there on your machine. An attacker could use it to generate certificates for almost every site and your devices would trust them. That's probably ok if only your machine trusts that CA since if the attacker rooted your box to the point that they gained access to that CA key, all is lost anyways. In a network with other devices - maybe even not under your direct control - that tradeoff looks substantially different.
I do have to set an environment variable so nodejs trusts it.
- have a proper worldwide domain
- obtain a certificate for that domain
- point the domain to local IPs in your network and use the certificate on the local server.
Doesn't change that you'd need to self-sign certs for .local or other funky domains.
More controversially I think you should give things globally unique addresses from the Internet's global address system but it's more important to at least give them names from the globally unique system even if you insist on using RFC1918 numbers.