← Back to context

Comment by tetha

19 hours ago

> He added that global SSL/TLS is a small part of eMudhra’s business contributing to about 1% of the global revenue

Good that we are now investing thousands of engineering hours globally to fix this.

Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.

Sorry, it's been a long week.

Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.

It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.

  • The number of awesome things we're going to get as a result of this is exciting to think about.

    My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.

    • Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)

      3 replies →

    • This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.

      6 replies →

  • > Yes, but in the end that's not their fault, and in a sense it's good that this happened.

    I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.

    That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.

    Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.

    • The leadership page is comedy gold ("Leadership That Inspires Trust"):

      > He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...

      > A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...

      > Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.

      https://emudhra.com/en-us/leadership

      ---

      The CEO guy had 45 years of experience and still managed to make a mistake that would get a high school student an 'F' on their hw assignment.

      2 replies →

    • Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.

      I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.

      3 replies →

  • Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.

    We've had to rework part of our application and certificate use because of this.

  • Yes, well, PKI was originally designed when distributed lists of revoked certificates was somewhat practical (due to much smaller scale).

    Now we are in a situation where we don't really have a practical and correct way to revoke certificates, but many, many things depend on pki.

  • > It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.

    No, this is not even remotely sensible.

    • Failure paths that are not exercised with regularity will inevitably fail in surprising ways when actually needed.

    • It really is. It's basically chaos monkey for pki.

      Code paths and procedures that aren't exercised regularly are broken.

      Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.

  • Honestly, that's not a bad idea at all. That's exactly the kind of thing that companies like Google do in disaster recovery testing.

    If you don't know if you can hot-swap your certs, you probably can't.

It baffles me that software on Linux/Windows/MacOS doesn't just use the system certificate store, then it would just be a single place to update. Instead we need to be update pip, UV, node, etc separately.

I can see the case on phones: the OS may no longer be supported by the vendor, so browsers shipping their own set makes sense. And on embedded there isn't (and on IOT there might not be) an OS, but then you should probably just ship the the single CA that you use, not all of them.

It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!

sounds likethey're making an argument for distrusting any CA for whom CA operations aren't a core part of their business.

I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA

But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem

  • As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.

    • Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken

      "no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.

    • This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.

  • Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.

  • It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.

    That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.

  • It's the attitude that riles me.

    This demonstrates how hard we are messed up in a CA revocation. But no need to rub it in like that, is there?