Hacker News Reader

Comment by tetha | Hacker News Reader

> He added that global SSL/TLS is a small part of eMudhra’s business contributing to about 1% of the global revenue

Good that we are now investing thousands of engineering hours globally to fix this.

Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.

Sorry, it's been a long week.

View on HN

Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.

It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.

> It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.

No, this is not even remotely sensible.

Failure paths that are not exercised with regularity will inevitably fail in surprising ways when actually needed.

It really is. It's basically chaos monkey for pki.

Code paths and procedures that aren't exercised regularly are broken.

Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.

Please elaborate. Conventional thinking goes that emergency plans must be exercised to make sure they’re actually effective.

The problem isn't revocation of end-user identity certificates. There's hundreds of millions of those floating around at any given point, so revocation happens very regularly. The issue is revocation of intermediates and roots, which involves invalidating tens of millions of certificates and often find themselves in embedded/non-web applications (financial, legal signatures, medical data exchange).

An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.

None of the test certs would be used for real, whether they're intermediate or leaves.

They'd be created with big red warning signs, installed into browsers and other places by anyone who wants to test, and then publicly revoked every month. Rinse, repeat.

You can do that, and I'm sure people already do. But you can't truly test things at scale. Are OCSP servers going to get overloaded? What about IoT devices and things out of support? You have deployments which configure their systems in non-standard ways, maybe some corporate manager decided to mandate cert pinning.

The original comment was to do a fire drill and revoke live certificates to force everybody to participate in the test.

You can have a fire drill and evacuate an office building, but you can't hold a full-scale forest fire/volcano/hurricane drill and evacuate 3.5 million people to test mass evacuation routes, and also you got the edge cases that don't participate in drills, like homebound seniors or hospitals, where disaster strikes.

> I'm sure people already do. But you can't truly test things at scale.

Do they, actually? I got the impression from the article that little revocation testing of any scale happens. If so, then even a limited test would be useful.

Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.

We've had to rework part of our application and certificate use because of this.

Kind of what Let's encrypt did for automating certificate rotation.

Let's call it "Let's Decrypt!"

I like it already.

> Yes, but in the end that's not their fault, and in a sense it's good that this happened.

I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.

That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.

Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.

Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.

I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.

> I want a single-file executable with zero dependencies.

Isn't this what Go programs are to a certain extent?

Yes. Get ready for Go programs produced by inexperienced programmers to do DST conversion incorrectly after the next rule change.

(Actually Go allows you to embed the tz database instead of forcing you to hardcode, so I hope this issue won’t be very widespread.)

There's a bunch of changes recently / coming up in Canada, so that might be good enough. I've started occasionally reading the tzdb mailing list archives for fun because of that.

The leadership page is comedy gold ("Leadership That Inspires Trust"):

> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...

> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...

> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.

https://emudhra.com/en-us/leadership

---

The CEO guy had 45 years of experience and still managed to make a mistake that would get a high school student an 'F' on their hw assignment.

It takes much less than 45 years to forget almost everything that was taught at high school

You say comedy gold. However in Vishwaguru Indian universe they are much vaunted South Indian Brahmins who anyone in the world accept that they are the best in anything they do. Hence they are pure gold.

The number of awesome things we're going to get as a result of this is exciting to think about.

My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.

Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)

Presumably just mitmproxy https://news.ycombinator.com/item?id=50189232 though this assumes the cert isn't pinned, which for IoT it may very well be (and probably should be).

You're probably right. Based on its MAC and some other quirks about it, I'm certain it's just an ESP32 inside. So if I end up not being able to bend the firmware to my will, I'll be able to bend the hardware to my will instead.

Use my pihole to make the domain resolve to some machine on my LAN, then use this glorious new mitmproxy feature:

https://www.mitmproxy.org/posts/releases/mitmproxy-12.3/

It’s a pity they didn’t name the switch something like —dont-use-emudhra.

PPMs > threshold, health insurance revoked

This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.

a high quality set of FOSS, community inspected "freemium clones" may do more for national security and individual privacy compared to yet another crypto library with zero novel mathematics just another reimplementation of specs...

I've had a PiHole running on a NUC for about 2.5 years. I adore it. It should be built into every consumer router.

My wife still plays her freemium games, the ads just don't load. She took out one of those games when we were sitting at the airport once. She was frustrated that her once-pleasant strategy game was suddenly crammed full of ads for boomer slot machine apps, and I had to explain that the app had probably always been that way.

That makes sense and seems like it might be a net positive. While she's waiting for her ad countdown, she can gaze into my eyes and realize how much she appreciates me lol

My girlfriend deliberately watches the extra ads to get in-game rewards, so I can't even pihole the whole home network unless I exclude her phone. (I find that most of these games work fine if you block ads, but they also don't give you the rewards for watching an ad that never loaded.)

Wondering if there's some way to 'load' it by triggering whatever it counts as engagement but still not serving it to the client. Claude, you have $3 left and daddy doesn't want to make mommy mad again today

Opus 5.5 will happily decompile .apk's and strip out the ads.

Honestly, that's not a bad idea at all. That's exactly the kind of thing that companies like Google do in disaster recovery testing.

If you don't know if you can hot-swap your certs, you probably can't.

Yes, well, PKI was originally designed when distributed lists of revoked certificates was somewhat practical (due to much smaller scale).

Now we are in a situation where we don't really have a practical and correct way to revoke certificates, but many, many things depend on pki.

It baffles me that software on Linux/Windows/MacOS doesn't just use the system certificate store, then it would just be a single place to update. Instead we need to be update pip, UV, node, etc separately.

I can see the case on phones: the OS may no longer be supported by the vendor, so browsers shipping their own set makes sense. And on embedded there isn't (and on IOT there might not be) an OS, but then you should probably just ship the the single CA that you use, not all of them.

It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!

Python shipping its own CA bundle is a criminal offense all on its own.

Don’t Java and Go do something similar?

Go does not. For Java, I would not look at their x509 handling for inspiration.

I wasn't implying inspiration, the point is that Python is not the only major language that does this. Node.js is apparently another. Rust crates like webpki-roots also bundle Mozilla's store.

There are reasons to do this, like support for minimal environments, consistent cross-platform behavior, simpler deployment, and application control over root certs.

sounds likethey're making an argument for distrusting any CA for whom CA operations aren't a core part of their business.

Part of it is on everyone else for not actually testing they key strength.

I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA

But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem

It's the attitude that riles me.

This demonstrates how hard we are messed up in a CA revocation. But no need to rub it in like that, is there?

Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.

As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.

The root CA should have been revoked then.

This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.

Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken

"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.

It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.

That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.

Reply on news.ycombinator.com