jaas

Let's Encrypt | REMOTE FULLTIME | letsencrypt.org

Let’s Encrypt is a free, automated, and open Certificate Authority.

Please submit resumes to: it-jobs@linuxfoundation.org

Systems Administrator

Our operations team has built the infrastructure needed to operate a secure, high availability, high volume certificate authority. There are high standards for the work that we do and the world is watching.

A candidate for sysadmin at Let’s Encrypt should have a solid background in 24/7 production operations and have experience with all the components of a modern datacenter environment. You should have at least some experience with security and stability monitoring, virtualization, firewalls, configuration management, database management, rapid provisioning and systemd.

Collaborating effectively with ISRG developers and the community is critical. You’ll be working with both fellow employees and our open-source community. ISRG staff live in various places in the U.S. and Canada, and we do the vast majority of our collaboration online.

Key Qualifications: Experience working with highly-available, internet-facing sites Experience in regulated environments Configuration management experience (SaltStack, Ansible, Puppet, Chef) Ability to effectively communicate and collaborate with development team Database troubleshooting experience Scripting experience Networking management experience (firewalls and switches) Experience implementing monitoring and logging systems

Nice to Have: Hardware Security Module (HSM) experience Coding experience Hardware SAN experience Experience using and managing security tools and vulnerability scanners

Good point, thanks. We just pushed a fix for that language, should show up live shortly.

It's possible, but a couple of points to consider:

1) Wildcards may have a net negative effect on total issuance if subscribers who were getting many non-wildcard certificates before replace them with a single wildcard. I'm not sure this will be the case, but it could be. I expect wildcards will help move many more sites to HTTPS - perhaps just with fewer actual certs.

2) Wildcard validation from Let's Encrypt will be restricted to DNS validation, and many people don't have the ability to automate modifications to their DNS records. That being the case, it will be a bit more difficult to validate for wildcard certificates than for non-wildcards.

We set that up before we were an operational CA and it has just never been a high enough priority to replace it. The team that would do it has a lot of other things to do. We'll get around to it at some point.

Good question. We are hoping to have a cross-sign from a widely trusted ECDSA root in place. It's non-trivial to make that happen, and it's part of the reason this feature is taking longer than we had hoped.

Let's Encrypt | REMOTE | https://letsencrypt.org/

Please submit resumes to: it-jobs@linuxfoundation.org

Systems Administrator

Our operations team has built the infrastructure needed to operate a secure, high availability, high volume certificate authority. There are high standards for the work that we do and the world is watching.

A candidate for sysadmin at Let’s Encrypt should have a solid background in 24/7 production operations and have experience with all the components of a modern datacenter environment. You should have at least some experience with security and stability monitoring, virtualization, firewalls, configuration management, database management, rapid provisioning and systemd.

Collaborating effectively with ISRG developers and the community is critical. You’ll be working with both fellow employees and our open-source community. ISRG staff live in various places in the U.S. and Canada, and we do the vast majority of our collaboration online.

Key Qualifications:

    Experience working with highly-available, internet-facing sites
    Experience in regulated environments
    Configuration management experience (SaltStack, Ansible, Puppet, Chef)
    Ability to effectively communicate and collaborate with development team
    Database troubleshooting experience
    Scripting experience
    Networking management experience (firewalls and switches)
    Experience implementing monitoring and logging systems

Nice to Have:

    Hardware Security Module (HSM) experience
    Coding experience
    Hardware SAN experience
    Experience using and managing security tools and vulnerability scanners

More stringent validation methods won't help with the ever-present possibility of private key compromise. So long as that's a real possibility and revocation is broken (which it clearly is), longer certificate lifetimes are a liability. Renewal needs to be automated so you don't care how often you have to renew.

Let's Encrypt will sign your ECC keys now, but we'll sign with our RSA keys. We'll likely have our own ECC trust chain some time next year.

Let's Encrypt doesn't directly produce any client software. ACME clients are built by our community and the community builds what they feel like building. If something hasn't been done yet, or isn't where you want it to be, that reflects community priorities, not our politics.

We'd love to work more closely with Microsoft. We've been talking with Microsoft and various Microsoft community members about building great support for IIS since before Let's Encrypt launched.

> I doubt the web will ever be fully encrypted. There will always be tons of people who don't care enough.

I don't think the Web will become fully encrypted by virtue of everyone caring enough to move to HTTPS.

The knockout punch for unencrypted sites will come when browsers make the decision to not load plain HTTP sites by default in order to protect their users. At that point, for the vast majority of people, the Web will be 100% HTTPS because they'll never see a site that isn't HTTPS.

We just have to get the encryption percentage high enough to allow browsers to finish the job.

A discrete GPU is a GPU that's not on-die with the CPU. A discrete GPU is usually something you stick in a PCIe slot.

A GPU on the CPU die, non-discrete, is often referred to as an "integrated GPU" or "integrated graphics." They're typically not very powerful, though they run common non-gaming applications just fine.

Let's Encrypt | Systems Administrator | Remote | Full Time

Let’s Encrypt is a free, automated, and open Certificate Authority.

https://letsencrypt.org/jobs/

Supplying an email address to Let's Encrypt is optional.

The fundamental problem with the legacy Firefox extension API is that its incredible power comes from putting incredible constraints on Firefox internals. Every stable API commitment involves some constraints, but the constraints from legacy Firefox extensions were out of control. They were killing the browser.

The "legacy" Firefox extension API is powerful because it reaches its tentacles into every corner of the internals of the browser. Keeping that API stable involves freezing a large amount of internal architecture and behavior.

Firefox needs to evolve if it's going to live, and it can't evolve in the ways that it needs to if so much of its internals are effectively frozen. It's a painful drag on development, slowing things down and preventing necessary changes.

I'm not even going to get into the problems with the API itself except to say that it had many, including widespread dependencies on synchronous behavior.

Mozilla's sacred-cow-level commitment to preserving legacy extension support is a big reason why Chrome beat Firefox on so many technical levels for so long. It ate up a ton of developer time, held off or killed critical architectural changes, was the cause of a lot of crashes, hangs, memory leaks, security problems... Thank goodness that's about to be over.

I'm sorry that you'll miss it but this is a really important step. Mozilla is doing a great job with WebExtensions, I suspect you'll be happy with how things shake out in the end.

The entire defense/post boils down to this line:

"... I ought to be more concerned with where the trend in memory is going rather than where it has been."

The problem with this line of argument is that it never ends. Developers who accept this line of argument are always aiming five years down the road, and thus always delivering a poor or substandard experience to their current users.

Your future users are not real, and if you're always targeting future users you're never targeting real people.

I've used these challenges in a few different contexts, they're great.

I taught myself rust by implementing solutions to these challenges, and I taught a CS class in which I had students improve their C language skills by implementing solutions to these challenges. Students got some crypto knowledge on the side!

This is a bit long, but I know the answer in detail because I'm the one who originally wrote to code to make 32-bit plugins work in 64-bit Firefox on OS X :) I'll never forget the moment I first got a YouTube video to play properly in 64-bit Firefox with 32-bit Flash.

It was much easier to do on OS X than on Windows. That's primarily because on OS X, at least at the time, Firefox was a Universal Binary, meaning there were complete 32-bit and 64-bit binaries for OS X in the same application package. We had been doing this for reasons I won't get into here, but they were unrelated to running plugins with mismatched bit-ness. That being the case though, we "just" (devil is in the details) started a 32-bit child process, which had all the same functionality (e.g. plugin code implementation) as the 64-bit process, and made sure that plugin API data going between the processes was always fixed length (e.g. int32 vs plain int) so there was no misinterpretation.

Mozilla could probably have done something very similar on Windows, but they weren't already installing both 32-bit and 64-bit binaries on every Windows machine. Doing that just for the sake of plugins would, at a minimum, have been a big bullet to bite. And there was/is no nice Universal Binary package equivalent on Windows.

Also, and this gets more nitty-gritty, the NPAPI plugin API on OS X was much easier to get working across two processes of different bit-ness. This was mainly because we had recently revised the OS X NPAPI (we called it Cocoa NPAPI or something like that) and got Adobe and many others to switch. The revised API was much easier to work with. The ancient Windows NPAPI was, and is, a total f*ing mess.

IIRC, in the case of Firefox the problem was not porting Firefox itself to 64-bit. That was done a long time ago. The problem was all of the 32-bit dependencies people had installed.

For example - if a user only had 32-bit Flash installed and they updated or installed 64-bit Firefox, they'd be annoyed that Flash "broke". That might not be the best or most current example, but it's that kind of thing that held up 64-bit default on Windows.

Publishing private keys is a violation of the Let's Encrypt terms of service. We are revoking these certificates.

https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016...

"LetsEncrypt issues SSL certificates by automatically verifying that you have ownership of the domain you claim you have."

Let's Encrypt does not verify ownership of domains. We verify control.

This is a common mistake, but the difference between ownership and control is significant so I'm just pointing this out for general educational purposes.

So your argument is that because rust doesn't solve all problems people shouldn't strongly consider replacing C code with it, even though rust avoids entire classes of critical bug types in C?

I happen to like rust, though I would not suggest that it is the only solution to the classes of problems it avoids. I would, however, suggest that people very strongly consider using languages that avoid the problems that rust avoids whenever possible. Like, literally any time you're considering writing C or C++, at least strongly consider rust or another much safer option.

You won't have confidence that your visitors see the content you serve as you intend because it can be modified in transit. Your visitors will be broadcasting at least the exact address of the pages they read on your blog, and are vulnerable to anything that gets injected into your traffic in transit (e.g. malicious script).

Every website should use HTTPS. It's the right thing to do. It's not hard to do these days.

Currently we're only planning to allow DNS method validation for wildcards. You'll have to validate the base domain via DNS, that's all. No HTTP or file-based validation option.

We decided not to offer HTTP-based (file-based) validation via randomly-generated subdomains for wildcards in part because if you're required to set up random subdomains you're modifying DNS to do that, and if you're already modifying DNS you might as well just use the DNS validation method.

I don't really understand what you're implying but here's some information that might help.

Let's Encrypt is a non-profit service and we don't use your data for our benefit. We work hard to collect and retain as little of your data as possible.

We do not require you to run software from us on your server. You can use any client you want. In fact, we don't even produce client software -- all Let's Encrypt clients are built and maintained by our community (the client we often recommend, Certbot, is an EFF project).

Yes

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused.

I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem.

OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our engineers observed a rather odd, but standard, web server behavior. When a server receives a request with multiple slashes one after another they will collapse them into a single slash. This caused our OCSP responder to consider requests that had this unusual encoding quirk invalid and would respond to with a '400 Bad Request' response. The fix seemed quite simple: disable the slash collapsing behavior.

Unfortunately, stopping this behavior surfaced a more serious issue. The AIA extension that we include in certificates we issue contains a URI for our OCSP server. This URI contains a trailing slash. According to RFC 6960 Appendix 1 an OCSP request using the GET method is constructed as follows 'GET {url}/{url-encoding of base-64 encoding of the DER encoding of the OCSPRequest}' where the url 'may be derived from the value of the authority information access extension in the certificate being checked for revocation'. A number of user agents take this quite literally and will construct the URL without inspecting the contents of the AIA extension meaning that they ended up with a double slash between the host name and the base64 encoded OCSP request. Before we disabled slash collapsing this was fine as the web server was silently fixing this problem. Once we stopped collapsing slashes we started seeing problems.

From our OCSP server's perspective a majority of the OCSP requests we were receiving were prepended with a slash and we were unable to decode them so we'd respond with a '400 Bad Request' response and move on. This coincided with a large number of previously cached responses on our CDN expiring, causing us to start getting hit with a large number of requests. Because we were responding with '400 Bad Request' responses we were setting explicit no-cache headers which meant we had a near 0% cache (CDN) offload rate and were hit with the full brunt of our OCSP request load at our origin servers. This caused our whole infrastructure to get bogged down.

Good news. Now they should add ACME support directly to nginx so it can get and manage TLS certificates seamlessly for you.

TLDR: Look at the IdenTrust numbers if you want to know the real Let's Encrypt numbers.

Not many people use Let's Encrypt's own root for their validation chain when they use Let's Encrypt certificates. The Let's Encrypt root is not propagated widely enough across clients.

We (Let's Encrypt) have a cross-signature from IdenTrust, that's what most people use because it's widely trusted. IdenTrust issuance is otherwise very small, so you can effectively consider the IdenTrust numbers in charts like this to be a proxy for Let's Encrypt.

-Josh from Let's Encrypt

I did a lot of work on Firefox's NPAPI implementation over the past decade. Improving that code was rewarding in that it had very tangible benefits for Firefox users but it was pretty clear that improving Firefox's code was only going to get us so far, and not far enough. The system is a mess and I couldn't be happier to see Firefox dropping support. I hope I can be the one to rip out the code altogether some day.

Head of Let's Encrypt here. Our team is looking into this and so far we don't see any evidence of mis-issuance in our logs. It looks like the domain in question, 'apple-id-2.com', was registered and DNS resolved for it successfully at time of issuance. Here is the valid authorization record including the resolved IP addresses for 'apple-id-2.com':

https://acme-v01.api.letsencrypt.org/acme/authz/uZGv2KXUJ6Hl...

We can't be sure why the reporter was unable to find a WHOIS record, we can only confirm that validation properly succeeded at time of issuance.

Update: Squarespace has confirmed that they did register the domain and then released it after getting a certificate from us.

Let's Encrypt | Systems Administrator | US and Canada | Remote

We are a free, open, and automated certificate authority. Our goal is to help create a more secure and privacy-respecting Web by driving adoption of HTTPS.

We're currently hiring another Systems Administrator.

Key Qualifications:

* Experience working with highly-available, internet-facing sites * Solid understanding of PKI * Configuration management experience (SaltStack, Ansible, Puppet, Chef) * Ability to effectively communicate and collaborate with development team * Hardware SAN experience * Scripting and coding experience * Networking management experience (firewalls and switches) * Experience implementing monitoring, security, and logging systems * Database troubleshooting and HA maintenance experience * Ability to use and manage security tools such as FIM, IDM, SIEM, vulnerability scanner * Experience in regulated and high-security environments * Hardware Security Module (HSM) experience is a plus

Please see full job description here:

https://letsencrypt.org/jobs/#systems-administrator