eMudhra ignored Mozilla Root investigation deadline; blames social media
bugzilla.mozilla.org
bugzilla.mozilla.org
> "There are a lot of questions here. As most of you can imagine, we have been subject to intense social media scrutiny and analysis by investors/customers over the last few days and it has made doing any meaningful analysis of the event pretty much impossible."
Mozilla Root Program's update, stating that eMudhra has not responded to investigatory questions within deadlines (and had provided an extension): https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c42
> "The Mozilla Root Program sent the attached questions to eMudhra on the 20th October at 20:31 UTC. eMudhra was asked to provide an initial response within 24 hours. After the deadline passed on October 21st, eMudhra asked for a further 24 hours to respond. As of the time of writing on Friday 23rd, Mozilla has not received a response to these questions."
I may get downvotes, but this just screams amateur hour.
Do some countries possibly value quality or responsibility as a desirable property more than others at a cultural level? (Is "German Engineering" a real thing?)
Why do 419-style scams seem to emanate from a few specific countries? Why is piracy on the high seas more prevalent in Somalia than the United Kingdom?
It is, but those generalisations are usually debatable and will then lead to nationalistic debates that are allmost always flame war style, with people shouting over each other who is from the worst nation. Which is why many flag anything, that smells like leading towards this.
So we should just close our eyes and not say accurate information, got it
I mean if I'm going to make really large generalizations about Software Engineering talent by country, Swiss people are the absolute bottom of the barrel maybe followed by the cheap outsourcing countries next, but ymmv.
I'm also sure i've worked with a lot more very good Indian engineers than German engineers here in America, but that might have more to do with who immigrates here.
Just by fact of population, a generalization may or may not hold more weight. Anecdotes are also tricky to really apply broadly: it's simply impossible for you to have encountered that many Swiss software engineers because few exist... few Swiss people exist. In 2019, there were over 300,000 Indian H-1B petitions to Switzerland's total of 202.
> I'm also sure i've worked with a lot more very good Indian engineers than German engineers here in America, but that might have more to do with who immigrates here.
Relative quality of life in Germany is objectively better in India by pretty much every metric, so there's an automatic/inherent bias - it's probable that more Indians wish to immigrate to the United States. (More than 10% of the Indian population doesn't have access to modern toilet facilities, for instance.) The competition for U.S. immigrant visas is far higher amongst Indians simply by the numbers, so there may also be a filtering function at play.
americans die like men -- by guns
except for the small children, they die like children being slaughtered in a school shooting. price of freedom amiright?
My comment was somewhat flip, but I wasn't surprised when I saw what and where.
How does that work exactly? Perhaps if they paid attention to their software instead of browsing X, they'd be able to get more done?
Everything this company says is ridiculous.
They seem equally bad at both.
Perhaps there would be responses that resemble the one here
I've always thought the "commercial CA business" was BS; it attracts these type of people
It relies on fabricated "authority" just like "ICANN". The later is rife with unethical practices and corruption. Why would CAs be any different
"pretty much impossible"
Translation: "We can operate a CA if we only have to communicate with so-called "tech" companies and our "business" remains mysterious to the public. We did that for many years, we are experienced. But we cannot operate a CA under public scrutiny from investors and customers"
This incident will not be enough to get rid of these phony "authorities" but maybe it's one small step closer
And the discussion mentioning wrap/unwrap testing kind of feeds directly into the fear that either:
1. These weak keys were created outside the HSM and then imported.
2. These weak keys were meant to be the wrap keys somehow. Like, if they hadn't paid for some of the wrap/unwrap features on one of their HSMs and so were using easy-for-humans keys to handle the wrap/unwrap operations so they could more easily move keys between HSMs.
Also depending on the FIPS configuration of their HSM, some functionality like wrap/unwrap might not be available
This should rarely happen if we are doing things correctly.
The moment the private key leaves the secure context of the HSM you should consider it to be compromised. Some HSM vendors have a way to effectively extend the secure context across multiple physical machines to address the edge cases where you'd be inclined to do this.
The problem isn't foreign companies; it's bad CAs.
man fox news has really done a number on people's brains.....
> only trust other US citizens with my security
I'm not sure what you mean, but this doesn't seem to be a coherent concept.
Famous examples include: Samsonite vs. Indian Competitors like VIP - Samsonite won the case.
Novartis vs. Indian Generic Makers. This went all the way to the Indian supreme court before Novartis lost the case. (And thank heavens for that).
Microsoft and Dell: They have won nearly all their cases in India involving unauthorized software and misuse of trademarks.
These are just the big names. There are dozens of smaller US companies.
All cross-border business is fundamentally import/export and comes with the same risk - that when I am wronged by someone in another country, I have no legal recourse. The exceptions to this rule (like airlines) are such enormous exceptions that there are literally international treaties (e.g. Montreal Convention) to enable it.
Nothing fundamentally US-centric about that point of view.
What you cannot reasonably expect is for the entire Internet to operate on a "these countries good, these countries bad" model, for many reasons, including that the citizens of your preferred countries are wildly outnumbered.
Anyone who is not in the US saying, on here, "I do not need CAs that are not my national ones" would realize the problem before even posting.
Trying to equivocate between Let's Encrypt and eMudhra is crazy, as is trying to argue that they should both be trusted globally by default by everybody. I trust LE because of their track record. I had never heard of eMudhra before this.
You say this, in 2026, when the EU is trying to move to an EU Sovereign Cloud Framework: https://commission.europa.eu/news-and-media/news/sovereign-c...
I have a deep problem with your usage of the word "need". EU citizens can make a free choice (or not) to trust US-based certificate authorities, and US citizens can make a free choice (or not) to trust EU-based certificate authorities. There are certainly many reasons why that should be the case. But in a world where we need to negotiate whether food safety standards in X economic block are sufficient for Y's citizens and vice versa, and this is literally the stuff of trade deals, the idea that US-based certificate authorities should be universally trusted because of little more than historical legacy is not the policy position of people who believe in free markets and free choices.
Can't use a computer then.
Or worse, you were billed incorrectly by some US-based service, their support chat loads Indian assets, and they threaten to hit your credit score and send debt collectors after you if you don't pay up.
No need to use international services like the internet.
Trusting a CA that uses 4 as the private key is risky. If it's a small CA that issued certs that you don't need to consume, it would be an unnecessary risk to trust it.
So you'd just end up with the big players (ex. Sectigo, who cross-signed these certs btw), who would then claim too big to fail should anything actually go wrong with them (see: Comodo).
Check InCommon Certificate Service.
If you want only to connect to the US net, stop connecting to the internet and use a US browser instead of a web browser.
They have that right though? There is no monopoly to single CAs for a given domain.
eMudhra intermediate CA compromise
https://news.ycombinator.com/item?id=50166579
1271 points | galadran | 3 days ago | 504 comments
--
Eight Bits of Trust: eMudhra's CA keys cracked in milliseconds
https://news.ycombinator.com/item?id=50177916
935 points | hkc | 2 days ago | 460 comments
--
eMudhra to exit global SSL/TLS certificate business
https://news.ycombinator.com/item?id=50195110
1111 points | bo0tzz | 1 day ago | 713 comments
[0] https://www.felonybench.com/
[1] https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...
I think we need to put Sam Altman in jail.
And I think it'll be an instructive experience for the industry and for the world.
Right now, it would appear that the strongest external consequences for fucking up a CA are strongly-worded posts in a web forum. I would hesitate to call the wind-down of a business unit a "consequence" in this context because it's the default outcome. Everyone was already playing that game. We need an additional layer of pressure to encourage good behavior.
It would seem that we have different definitions of "wind-down". Everyone in the business operated with this consequence already on the table and it did not seem to provide the desired outcome.
Like it or not, CA infrastructure is global, and there’s no single set of laws that can or should govern it.
No, the strongest consequence is their stock value. It has gone down a lot and if they continue not following their agreed obligations as shown I hope that trend will continue.
Sure browsers are safe, but countless other internet softwares aren't (curl, python, pip just to name a few I know off the top of my head). Pip has a big problem because even if the OS distrusts the eMudhra certs, it still does because it has its own trust store. This will be an attack vector (that now everyone knows about) for a while.
And guess what: this already exists. DO-178C for aviation, ISO 26262 for automotive, IEC 62304 for medical, etc. But these standards are specialized to the industry because, no surprise, there is no appropriate one-size-fits-all software development approach. You can’t just say “software must be bug-free” and leave it at that. That’s been shown to be impossible. Even formally verified software can have spec bugs.
There’s already formal standards that CAs must meet, and they get audited too. Maybe those standards aren’t rigorous enough. But even with those standards, what authority is going to punish a CA who breaks them, other than browser/OS vendors? If it damages the host nation, you might see something. But if the host nation is complicit, and used the rogue CA for intel purposes—or if the company has political connections—then I don’t think you’re getting the punishment you want.
It makes sense to have these heavy standards when you’re talking about software whose failure could immediately cost actual lives. I’m hesitant to expand it elsewhere as it could threaten the viability of open source and software freedom in general. Nobody should ever need a license to run a compiler or deploy an application to their own server.
Also, a purely software-focused standard doesn’t make sense for CAs, as they also operate and maintain infrastructure. The failure with this CA was a process failure.
I probably emphasized software too much in my reply, but that’s usually where I see people call for standards to be forced on the “tech” industry.
My opinion is that all the process in the world won't help if the people doing it don't care about the outcomes.
Why aren't SSL certificates tied to something like DNSSEC where a CA is responsible only for one tld? Since domain ownership is enough to get a certificate signed anyway, that wouldn't change security properties.
(2) Because DNSSEC is a nightmare to deploy. Leaving the IANIX-type operational stuff aside, it doesn't work on all network paths; plenty of middleboxes freak out when they see DNS packets with DNSSEC records in them. People see this and want a best-effort system that uses DNSSEC in the majority cases where it does work, but now you have a downgradable, strippable security protocol.
(3) eMudhra is being murdered by the root programs as we speak, and when they're gone all we'll remember them for is as a WebPKI incident. The WebPKI itself will continue to function and nobody will care. That can't happen with .COM or .CO.UK or .IO; there's no accountability for those registry operators at all, let alone transparency logs to track what they're doing.
As such, DANE would not add more issue here;
DNSSEC is easy to deploy (of course, you need to know your work, which is rarely the case). Legacy-first mindset is not really productive;
That's is correct, but on the other hand, incorrect: as a .com registry, I can:
- fake the CAA records
- fake the DCV records
- make any CA sign any certificates
This is more than twenty-five years after DNSSEC standardization commenced in earnest (counting from the bis protocol).
I completely agree that the tooling is questionable, and that the protocol is meh, but in this case the monitoring noticed the problem yet humans did not notice (or get) the alert.
Who? Companies whose audiences are certain kinds of corporations?
"Sectigo has completed the revocation of the affected cross-signed CA Certificates. A new CRL and OCSP responses have been published and are making their way to our CDN edge nodes right now.
We will continue monitoring and evaluating this bug for further updates from Emudhra."
Can't wait to see all the exploits this will enable.
You know very well that's never going to happen.
So tooling bears some of the blame, unless I’m misunderstanding the mistake that was made.
Doesn’t seem they’ll give a timely technical response, which keeps all of us guessing about.
If eMudhra just don't comply with Mozilla/Chrome, is there anything they can actually do?
> There are a lot of questions here. As most of you can imagine, we have been subject to intense social media scrutiny and analysis by investors/customers over the last few days and it has made doing any meaningful analysis of the event pretty much impossible. The event happened on April 30 2025 and was done by set of people who have worked in the PKI industry for several years and worked with us for a while. There is a lot of digging we need to do to understand the circumstances, why, what, how to share a meaningful report that can hopefully benefit the community. I do not know why some community members think we wont be getting back on this bug. When we signed up for the browser program trust, we also signed up for disclosure obligations to the community which we will uphold.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c39
--------------------
ETA- HN rate-limiting won't let me reply, but to the commenter asking how all the pressure could delay a sober analysis:
I've never been involved with a mess like this, but I can well imagine it's difficult to convene the necessary people for a dispassionate RCA when your bosses are furious and investors are screaming for scapegoats to fire.
--------------------
Edit2: Still ratelimited for the downvotes, but in response to @andor:
You're right, some of the questions are fairly straightforward and should be easy to answer -- I would argue they already did provide an initial response here: https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c11
But there's 12 separate questions in this comment alone https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 many of which are demanding detailed information and statistics that would need to be reconstructed from audit logs.
I imagine lawyers are involved because whatever answers they give will inevitably wind up as evidence in ensuing lawsuits. I'm sure anything publicly released needs to be cleared through several layers of review.
Something that could have been fully analyzed with those people in a couple weeks ending up taking two years with an independent outside auditor.
The independent auditor would have been needed either way, but if we gave the original crew the opportunity to respond, THEN fired them, the outside auditor could have just verified the story much more quickly.
That just sounds like a management failure. You don’t actually have to cave.
The only reason there would be to fire people immediately would be if it would have been done deliberately with malicious intent but in this case there is no evidence of that.
Scape goats work better when they can't defend themselves.
The trust was gone and it's good that they closed up shop, don't get me wrong. But it's not like anyone who knows IT could create certificates in their name, in this case the impact is really a lot bigger. Especially to the world as a whole, the DigiNotar issue mainly affected users in Iran.
In any sane business, the people handling the screaming investors are different from the people handling the social media mess, and they are also different from the people doing the RCA.
That said, given the mess, I wouldn't be surprised if eMudhra only had one employee responsible for all those jobs...
Such a business will not be long operating a CA, then: these sorts of requests are common and CA operations are done openly in public with all accidents openly admitted in plain view. If management can’t deliver an environment that prioritizes compliance over rage and blame, then termination of CA acceptance for the company they manage with such irrationality is a successful outcome for the process. I respect that there may be opposing corporate and/or cultural pressures, and I imagine that’s why they’re voluntarily exiting the CA business, but they’re at risk of having all issuances distrusted retroactively if they think that gives them the leeway to reply late.
For example, further up in the bug tracker, they stated that there was an error in a script. If they were able to find an error, why can't they provide the script?
This exposes how poorly they are prioritizing the response. They look like a deer caught in the headlights. Stop looking at the headlights and get moving.
The comment you link to is from 19th. How could that provide an initial response to a mail send to them on the 20th? That does not make sense unless they have a time machine.
> But there's 12 separate questions in this comment alone https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 many of which are demanding detailed information and statistics that would need to be reconstructed from audit logs.
But that is not the set of questions that were ignored for the deadline.
> I imagine lawyers are involved because whatever answers they give will inevitably wind up as evidence in ensuing lawsuits. I'm sure anything publicly released needs to be cleared through several layers of review.
If they want to lawyer up, they should better be removed from trust stores where trust is more important than admitting the absolute minimal amount possible to lawyer up for lawsuits.
Or they could try to regain some trust, but it looks like they don't want that as a closer look might find more issues, like the other post about their broken login process.
Why? They've already announced they're quitting the CA business. I assume their priority is avoiding getting sued into bankruptcy. Playing along with Google/Mozilla's star chamber can only harm them.
I'm lowkey impressed they've even said "we also signed up for disclosure obligations to the community which we will uphold" because it's all downside risk with zero benefit for their company.
This is so bad.
They're asking us to believe that a script flaw, on this specific day, for critical keys, silently generated a second, weaker key pair, and that the wrong pair was then used for the CSR because the labels looked similar. And "used by mistake because the labels were similar" means nobody checked the CSR's public key against the intended key before the root signed it. Worse, they had a direct signal months ago in Bug 1970728 their CRLs failed verification, and by their own account the fix was to re-sign with the flawed keys rather than ask why the "correct" keys didn’t match the certificates.
--
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)