Comment by rdtsc

20 hours ago

> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business

Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.

> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal

Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.

It was "multiple human errors". But yeah no matter what, in any case or exceptional circumstances, this thing was not supposed to be screwed up.

On Mozilla’s tracker, eMudhra said the keys were created on 30 April 2025 in a key generation ceremony, a recorded procedure in which keys are produced inside hardware security modules (HSMs), tamper-resistant devices built to hold keys. The keys were meant to be moved to data centres in the United States and Europe for customers needing data stored in those regions. To move them, the company added a step to ‘wrap’ each key, or encrypt it for transfer, and ‘unwrap’ it at the other end. eMudhra said a flaw in the software script created a second, weaker set of keys during this step, and because the two sets had similar names, staff used the weaker ones by mistake. It said human errors occurred at multiple steps.

source:

What's the eMudhra security scare that has rattled the internet?

https://theprint.in/tech/whats-the-emudhra-security-scare-th...

  • These "wrapping" keys were not "weaker". They were completely unfit for purpose. They have not explained why they used a 3-bit key for this purpose.

    If they go to the trouble of wrapping them, use real keys. Or don't do it.

    Also, it's a big "human error" to not notice this for a year and a half.

    • I have to imagine that the 3 bit numbers were slots/indexes on the HSM, and rather than actually retrieving the key(or maybe it did), the slot number got used as the key. I can't otherwise imagine how they'd generate such low number keys...

      4 replies →

    • Having worked around these types of devices and their interfaces, I doubt they intended to create weak keys.

      It is more likely they had a script which placed a parameter in the wrong field, due to something like using positional arguments. Or they misread the documentation for some interface.

      Still incompetence. And I would love to see my theory confirmed or refuted.

      7 replies →

    • Yeah. I can't think of any development process where you wouldn't simply use a production grade key in development. In fact, that's exactly what we do in development with snake oil keys.

      Generating a key with "4" was probably done as a joke, but also is insane to have ever lived outside some devs computer.

    • > These "wrapping" keys were not "weaker". They were completely unfit for purpose.

      Oh come on. They clearly didn't intend to use 4 as a private key or to wrap it. They had just purchased millions of dollars of HSM's to store them. It was something more subtle than that.

      Have a little imagination. The transfer procedure sounds bespoke - possibly just a shell script using openssl to encrypt things at one end, and decrypt them at the other. Let the man who has written 100 shell scripts without a bug in one cast the first stone.

      But not so subtle that good software engineering procedures wouldn't have picked it up. In fact today just an LLM review of the shell script probably would have caught it. The Internet Death Sentence was warranted - because poor software engineering procedures are unforgivable in a CA. But those procedures and policies aren't easy or cheap to come by. If they were just "don't chose 4 as a primary key" my grandkids could do it.

    • Unwrapped extraction is the kind of thing security by checkbox covers and process review might catch so it is never clear in a case like this that some party with the capability to influence the process didn't want keys intentionally weak to emulate unwrap.. Then someone else may have introduced the swap of keys that would never have been seen with keys that would (be forever) in a public scrutiny DB.

    • I don't think it's implied the wrapping keys were the weak ones. More like, the script that ran the wrapping still had the fast "debug" generation of a pair of keys to test the wrapper on enabled, for example.

Their complaint about social media reeks of "and we would have gotten away with it, if it wasn't for those meddling kids". [1]

[1] https://www.youtube.com/watch?v=hXUqwuzcGeU

  • We deeply and sincerely apologize for getting caught and not being able to cover it up.

  • This is pretty normal in India unfortunately. Crony capitalism is the dominant business engine in the country, so pretty much any significant business, esp in regulated sectors like govt mandated services(which is the bulk of eMudhra's operations) is impossible to set up without significant bribes being paid.

    As a consequence, these businesses are very well protected from normal market forces and are pretty much above the law. Had this incident not been an international one, they had a very good chance of getting away with it, and the people reporting this would have landed in jail.

    • This is 100% accurate. State run media would have squashed this article if it was not for an international incident response. They have federal laws that prevent such disclosures because it affects the national image and could cause social unrest.

    • Since you seem knowledgeable about this topic.

      Would you say the system forces people to be like that or people are the reason why the system is like that?

      Would you do business (i.e. trust) with someone that comes from that system?

      3 replies →

    • >capitalism is the dominant business engine in the country,

      The government _mandates_ that certain services be used. In which world does this rhyme with "capitalism"?

      2 replies →

  • It does, but I wonder how much of that phrasing is cultural.

    When working with different businesses across the globe, I’ve noticed Indian correspondence to incidents tends to be more a show of strength even if it’s transparently obvious that there’s no merit to the content.

    Whereas western European correspondence is often more apologetic and empathetic. Even if it might sound insincere to those affected.

    But it’s all just words at the end of the day. It never means much unless the business is willing to back real change to mitigate similar issues in the future and resolve undisclosed existing problems. However there’s no way for us to know if this is happening or not.

    • I don't know if it's really specific to one culture. Look at US companies after they go through a lengthy legal saga, it's obvious they screwed up, they're paying fines or having to adhere to court-imposed consequences. Their press releases:

      "We admit no wrongdoing. We are only settling to move past this. Our business is strong/secure/competent."

      It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.

      2 replies →

Their OTP solution is bruteforceable as well: https://mastodon.catgirl.cloud/@49016/117475921584308861

  • This becomes an inverse Russian roulette: only one of six chambers is not loaded.

    Waiting for a release like "... the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP 1FA place in our business. In view of this, after detailed deliberations our board has concluded that withdrawing from global OTP 1FA is the right decision for eMudhra and its shareholders"

  • Link is broken, gives some randomly generated text when using Brave on Android. Their AI bot detection is not very good.

    • > fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:

  • It's not clear what you have linked there.

    • Paste from mastodon:

      > fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:

  • "However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP's place in our business"

Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.

  • Then indirectly the security and trust of the entire Internet rests in the hands of a few government babus in India... that's scary.

    • Their Internet registry people also weaponized AS0 (zero) as a method to take networks off air via BGP RPKI for the ranges they're responsible for. There's been a number of situations where multinational companies who operate in India who are forced to use IPs from the India RIR have been fat fingered by some staffer who makes changes then goes home for the day: meanwhile a multinational bank is down for 24 hours within the country because said agency doesn't believe in 24/7 support.

      Typically they do this for non pay but they've dropped the hammer multiple other times incorrectly (and good luck getting a RFO out of anyone there).

    • For good or bad, thats by design. These is a significant portion of physical infrastructure associated with internet data flow. The only challenge governments face as of now is being able to segment the traffic to their wishes in its entirety.

      Companies already control all the traffic flow on their infrastructure using end-point security solutions, the same can be extended at ISP level, and then Country level

  • Regulation like this makes sense, but then the government should also srutinze these companies

    • Well Indian regulator must have visited office canteen to see no unapproved food was served there.

      Only when fully satisfied they'd have given license to operate this security business.

"out of proportion to SSL/TLS’s place in our business" is an insane comparison. They are saying they care so little about SSL/TLS to begin with that it's not fair.

It would be like me practicing surgery 1 hour a week and saying "well surgery only has a very small place in my 80-hour work week" in response to flagrant failure of medical care.

  • "We are bad at this job, but we don't do it often so don't think we are bad at other jobs".

    Yeah, the proportion if anything is too small. This is the sort of company that should not be doing anything related to security.

This is what I found regarding how it happened, but still not a full explanation - but there was some name confusion and obvious incompetence.

>According to eMudhra, proper strong keys were created in the HSM at the key ceremony on 30-04-2025. The script had been changed to test wrapping and unwrapping keys, because the intermediates were to be moved to data centres in the US and Europe, and a bug in that change created a second set of weak keys. The keys had almost identical names, and the weak ones were used for the certificate requests. The same request was signed by eMudhra's own root and sent to Sectigo, which cross-signed it. The strong keys never made it into any certificate (Bugzilla).

https://www.fairssl.se/en/articles/emudhra-ca-noegler-knaekk...

  • I wonder. Say I wanted to prevent something stupid like this from happening in the future. What would be a sensible check? The higher level authority doesn't see the private key itself, right? Just the certificate request. And I wouldn't want someone to manually inspect the keys to see that it looks random-ish. Perhaps there could be a small check script added to the generation of certificate requests that says "hey, your key a.) is mostly zeroes, b.) appears to be ASCII text, or c.) seems otherwise extremely non-random, please use -f to continue anyway".

    • I don't know if it will work on something as small as a private key, but you can compress the key. If the compressed key is significantly smaller then the original, it lacks entropy. Pure random data is mostly incompressible.

      Note that 32 0x00 is a possible output from a pure random generator while still being quite compressible, you would probably still want to throw that one out though. After buying a lottery ticket.

Russian Roulette is pretty accurate if not an understatement.

If they expect people to still trust them on their other services they have obviously no clue on how Trust works, meaning they shouldn't be in the game in the frist place.

As a trust provider you get one chance. Squander it and you're out. That is how it should be.

Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.

In any legal dispute involving them, Exhibit A will be "look at how they ran their SSL/TLS business". That won't go well.

They got their image hurt, and pulling the culturally appropriate response here which is to never apologize and to never accept fault. Instead quit/resign and say that other stuff is just fine. It's really not fine, and you can bet that the same corporate philosophy that made the CA error will have serious security errors in their other products.

I think it doesn't matter what they say because they'll just get away with it. It's as simple as that. They'll get away with it.