This is almost certainly related to some sort of Russian cybercrime investigation. If you ever read Krebs or peruse some of the seedier Russian forums, xmpp.ru will sound familiar to you, because it is. Not imputing anything to the operator, that's just the nature of operating an anonymous service.
Drug gangs widely use XMPP as a secure communication channel in combination with Tor. Hydra (the darknet market) has been taken down by German police in 2022 after trying to extend their "service" to the EU; something similar might or might not have happened here.
This criticism/argument was, in fact, made against domain validation (DV) when it was first introduced by earlier CAs. It's not an unreasonable criticism, but the economics of PKI mean that we would have much, much less encryption and authentication on the web today without DV.
DV represents a security trade-off, and Let's Encrypt took this and ran with it, with the net effect of much much much much less web traffic interception overall, albeit with known non-Let's-Encrypt-specific vulnerabilities to attackers who can manipulate infrastructure sufficiently.
As other people in this thread have noted, there are also other mechanisms that can help mitigate those vulnerabilities. Maybe we can come up with more over time!
> This shows that Let's Encrypt security is a joke because now any large national ISP can use same MiTM to issue a certificate for any site hosted within a country. The SSL infrastructure is completely compromised.
the information available right now are too vague to come to a conclusion this bold.
Instead, I find it something like the following more plausible:
jabber.ru and xmpp.ru seem to use "exotic" DNS servers (at least as I checked right now).
The moment you're able to write a (TXT) record for some domain name, you have proven to be eligible for getting an SSL certificate for that domain name.
For high risk targets, consider layering an additional auth mechanism that doesn't rely on trusted CAs: Tor onion services, SSH, or Wireguard.
> All issued SSL/TLS certificates are subject to certificate transparency.
+1. crt.sh has RSS feeds for this.
> Limit validation methods and set exact account identifiers
Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?
> Monitor SSL/TLS certificate changes on all your services using external service
+1. High-risk targets should be aware of what certs are valid at any time, and be checking for those.
> Monitor MAC address of default gateway for changes
A more sophisticated attack could preserve the MAC address.
> "Channel binding" is a feature in XMPP which can detect a MiTM even if the interceptor present a valid certificate.
>Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?
Author of ACME-CAA (RFC 8657) here. ACME-CAA can mitigate this because you can put a unique identifier for your ACME account in the CAA record, so it is not possible for an attacker to do this unless they can get your ACME account private key (or coerce the ACME service). This assumes you have DNSSEC-secured nameservers, of course, otherwise DNS requests can potentially also be intercepted when queried by the CA.
See RFC 8657 for a full list of security caveats. The RFC is designed to be more readable than most.
I usually brood on blogposts for days before publishing them since I care a lot about getting things accurate, but this is a bit of the moment, so here goes. Always happy to get feedback by email or IRC: https://www.devever.net/~hl/contact
Wuuut, I missed they finally enabled it. Everyone who has infra MitM in their threat-model should look into and enable this.
Also, thank you for your work as well as popping in here.
Am I overstating it by seeing it as one of the most important milestones for Internet security in general over the past few years?
Do you have a take on DLT-based systems and if this is something that is (or could be) seriously discussed? It seems to me that the issues we have with PKI and certificate transparency could actually be mitigated very well if blockchain was seriously considered.
Interesting and I can see some of the benefits, particularly in preventing DNS controllers from going to other CAs and making new keys, but... this all seems like a weird run-around for the MITM part.
Couldn't you exchange public keys with LetsEncrypt (in the web UI) and encrypt the response so you can't be MITMed? Why is http even an issue?
> All issued SSL/TLS certificates are subject to certificate transparency.
Until the nation state tells a CA within their borders to issue a cert without publishing the CT logs, along with a gag order. If they don't comply they go to jail. You think Billy Bob Sysadmin wants to go to jail to protect a russian jabber server?
The malicious certificate either won't have a SCT (which in itself would be highly suspicious and cause some clients to reject the certificate directly), or it will have one but it won't be recorded in the log.
In the latter case, the malicious certificate + any CT log issued after that certificate should have been included are evidence of the attack that is easily verifiable and would likely cause browsers to drop the (edit) CT log unless provided with a very plausible excuse that isn't "we complied with a court/gag order".
Also, this would require the police etc. to compel 2-3 different entities: The CA and two log operators (I think one of them could be the CA itself).
Using different logs than normal would stand out, so it'd typically be two specific log operators that would have to be compelled to hide this. The US may get lucky and have jurisdiction over all of them, other countries are much less likely to.
If the certificate isn't logged in CT, it's not supposed to be trusted. That's the whole point. If it is logged, then the site owners would notice (Cloudflare, for example, has a service that emails when certificates are issued for your domains).
Just to be clear (with better details already given by other comments), addressing this is specifically part of the threat model for Certificate Transparency. You could argue that CT policies should be strengthened in various ways, but the threat model does intend to address detecting the compelled misissuance scenario.
No, the German thing would have been to have the printout of the telefaxed scan of the court order collecting dust because the scanner in the receiving department is broken...
Their theories are interesting and would explain how they obtained the certs if true. Perhaps the take-away here is to have multiple probes on multiple providers checking all of ones TLS fingerprints and alerting on unknown fingerprints and then checking the certificate transparency log or lack thereof.
I've added "Could you prevent or monitor this kind of attack?" section.
There are several indications which could be used to discover the attack from day 1:
* All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
* Limit validation methods and set exact account identifier which could issue new certificates with Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (RFC 8657) to prevent certificate issue for your domain using other certificate authorities, ACME accounts or validation methods
* Monitor SSL/TLS certificate changes on all your services using external service
* Monitor MAC address of default gateway for changes
Nice addition. Me personally being the paranoid type, I don't trust the transparency log for monitoring but rather for writing up the root cause analysis. Reason being is that if someone can legally compel Hetzner and/or Linode and/or LetsEncrypt to do or not do something then the same entity can compel the certificate transparency site to ignore something. But you covered what I would do and that is to have multiple nodes doing active monitoring of TLS changes using an external service. That service being openssl s_client in my case.
In the case of Jabber it might be interesting to add some monitoring in the application that uses a cryptographic signed payload to list all the valid fingerprints and send an alert message to the server if something odd is happening like public key pinning but without a hard-fail. That list could be pre-loaded with new certificate fingerprints prior to being deployed. If the oddness is confirmed then perhaps add some way to tell the clients that certificate is likely forged. That way both server operator and client are aware that something evil this way comes.
[Edit] - Looks like based on the comment from MattJ100 that some Jabber servers and some clients can already do something like this.
I mentioned this in the operators channel just now, but also worth noting is that channel binding (e.g. SCRAM-*-PLUS in combination with RFC 9266) also mitigates this attack. Essentially the connecting client is able to detect the mismatch between the certificate or handshake the server thinks it is presenting, and the one the client actually sees.
ejabberd and Prosody support it, and a number of clients too.
DANE (with DNSSEC) may be an option to look at too, with public keys in DNS. though probably client support isn't universal (i have no (recent) experience using jabber). the CA was one weak point here.
checking for new public keys in certificates when keeping private keys between renewals would also help.
i suppose any storage and possibly ram should also be considered compromised? e.g. private keys stored on disk without encryption, but also by being read into memory. especially on the vm.
Let's encrypt (and other CAs) should not allow validation using unsecure schemes like HTTP. Especially for sites that have valid certificate from other CA.
I was a bit curious about the MAC address since it isn't locally administered which indicates to me that someone likely deliberately chose this range of addresses.
Googling for '"90:de:01" linode' and '"90:de:00" linode' indicates that addresses from these blocks have been assigned to other linode VMs recently. I sadly don't have a linode VM of my own to compare with right now but it would seem like the traffic has been routed to another VM on linode infrastructure
This is baseless speculation, but I'm assuming Jabber is being targeted as it's famously used on darknet markets for drug trades (or other illicit activity).
Goes to show that you should never just trust "it's encrypted, bro". You need to PGP your messages at the very least.
Is PGP crackable by quantum computers? Will there be hardening against those kinds of attacks in the future? Since, if the messages have been hoovered up in encrypted form, it's just a matter of time until they get decrypted.
And this appears to be done for just about all web traffic they can get their hands on... see https://en.wikipedia.org/wiki/Utah_Data_Center
Jabber/XMPP has had e2e encryption for at least like 10-15 years. I used to use it with even my normie friends back when Facebook/Google Talk supported XMPP and you could use pidgin, kopete, etc.
Obviously securely exchanging keys with an anonymous drug dealer over the Internet is error-prone though...
PGP certainly has its problems, but isn't really special compared to other similar things. The big advantage that PGP has is that it is a stable and well known standard. There is a tendency to imply that it is insecure in some way, but no real evidence seems to exist to that effect.
>End-to-end encrypted communications, such as OMEMO, OTR or PGP, are protected from the interception only if both parties have validated the encryption keys. The users are asked to check their accounts for new unauthorized OMEMO and PGP keys in their PEP storage, and change passwords.
The attacker would need to set up a separate MiTM for the particular E2EE scheme used. Some of the XMPP clients I have encountered will not let you use a particular cryptographic identity unless you have explicitly claimed to have verified it.
Still a good reminder. If you have not seen and dealt with a ridiculously long number (or equivalent) you have not achieved end to end.
There's one thing I can't understand in this story: if that's lawful interception, why Hetzner and Linode bothered to set up MitM interception with different LE certificate and key, rather than extract the TLS private key directly from the RAM and/or storage device of the VPS? Even if this is a physically dedicated server, they can extract the private key from the RAM by dumping the RAM contents after unscheduled reboot. Extraction of the private key isn't visible in CT logs, much more stealthier, practically undetectable.
There's also a possibility that one would be a "search" and the other would be an "interception" with different levels of approvals requested, but I don't know what the current legal situation in Germany is right now.
On a physical server, couldn't you just hotplug a PCIe card in there and DMA out any data you are interested in? Something like a network card with firmware specifically for the purpose should do it. It sounds so much a standard thing for law enforcement that I imagine such equipment should be available off the shelf?
I wonder if there is a legal way to demand Hetzner/Linode comments on this situation. Likely, the entity behind the interception is some government agency or police.
there isn't even any guarantee that the wiretapping was done through them instead of e.g. the carriers which pretty much in any country have since decades been forced to help with lawful wiretapping...
Carriers meaning the interconnect providers eg Level3, Cogent etc? How would this intercept be implemented in practice? Surely it'd be much easier to add a node as close as possible to the origin host, i.e. within the Hetzner network, rather than redirecting traffic from the outside with some sort of BGP hijack?
I think the observed TTL 64 means the interceptor is on the same segment? (of course unless they have set it to e.g. 66 at the interceptor that is 2 hops away, but I guess if they were to mangle TTL, they would set it to the original value to avoid detection)
one issue is that most hosting providers don't display ssh host keys anywhere in the control panels so unless you have a... lot of fun (and spend time) with the virtual console... (or maybe a custom cloud-init script?) you're SOL for the very very first connection
At DEFCON, fifteen years ago, MITM'ing TOFU ssh connections was what you did when you were bored in between talks. If you wanted to capture a new first connection, just keep dropping connections until they try a different host or device.
I don't really get why people use ssh at all if they aren't using certs. Might as well use telnet.
I've always thought a solution like LetsEncrypt would be an ideal backdoor for LEAs wanting to circumvent SSL/TLS.
The same thing applies to VPN services. Why bother trying to develop tools etc that can decrypt traffic when you can just have people send their traffic to you?
Wouldn't be surprised if this happened in Russia; but not in Europe. Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability - doesn't this mean that large national ISPs can easily issue ceritificates for any site hosted within a country?
> Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability
They can’t use HTTPS if you don’t have a certificate yet
>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023
Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?
It's fairly common for people to obtain multiple certificates for different machines or services, so they can be selectively revoked and they don't have to share keys across machines.
More use-cases:
- You might obtain a new certificate, but deploy it gradually, so you want the old one to remain valid while you do that.
- One certificate may cover different sets of domain names. If you have a certificate for "example.com, foo.example.com" and then request a certificate for only "foo.example.com", should the earlier one be revoked? (leaving "example.com" without a certificate).
> Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt?
it commonly used in a "normal" way all the time
- e.g. when there are multiple data-center for the same domain (e.g. using geo-location based routing) it's a good practice to give them different certs so that if you need to revoke one the operation in other regions is unaffected
- or when rolling over from on cert to another
- or when moving certs into hardware security keys/module (HSK) you preferably do have one per HSK (so that if e.g. hardware breaks and gets replaced you can just revoce the cert for the affected HSK module not all of them), you also normaly do not keep backups to make sure it can't be leaked at all (as long as the HSK isn't hacked which is normally quite hard)
- or losing access to a cert (e.g. in the case above a HSK breaks)
Lastly the whole CA system is in the end designed to provide good security for the industry while having the backdoor of issuing certs the legal organs to allow the police some degree of wiretapping (oversimplified, it's slightly more complex then that).
You should always have more than a single certificate for your domain honestly.
Cloudflare for example, tries to optimize certificate delivery (and have backup certificates available for you just in case a CA needs to revoke theirs).
Also, on distributed systems its less safe to share private keys between the various frontends.
This is actually a great suggestion and ACME providers should provide it as an opt-in feature via CAA record. Not even the provider having access to system memory could issue a mitm cert without you noticing.
You could sync certificates across hosts for this purpose, though. The advantage of multiple certificates is being able to revoke a subset of certificates if you can determine only a subset of your hosts have been compromised.
> Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?
First, you want to have to have some leeway so you don't need to rotate certs at exact second the old one expires
Second, you might want to have cert-per-server rather than cert-per-domain, as that's frankly easier to implement vs having common store for certs+key
Are there any services that monitor the certificate transparency logs and send an email when a new certificate is issued for your domain? That would alert you to this kind of MITM
As mentioned there are services for doing this (I've been using for example the free service from Facebook [1])
Since the certificates are nowadays renewing quite often, it's very easy to just end up ignoring these notifications. To really get the benefits, I think you should somehow combine these with logs from the legit cert renewals and only alert when something strange pops up.
Indeed, a CT monitor which sends alerts about legitimate certificates is almost useless due to noise. My service, Cert Spotter, provides an API endpoint[1] which you can upload your CSRs to, so you don't get alerted about certificates using the same key as the CSR. The open source version of Cert Spotter can invoke a script[2] when a certificate is discovered, and the script can cross reference against a list of legitimate certs.
If the CA doesn't log a certificate, the certificate won't work in CT-enforcing clients (Chrome, Safari, Edge) because it lacks receipts (SCTs) from CT logs.
If a log fails to publish a certificate despite issuing a receipt for it, Chrome's SCT auditing infrastructure can detect that, as it did recently with the Nessie 2023 log: https://groups.google.com/a/chromium.org/g/ct-policy/c/5x1A6... (this was also detected by monitors operated by myself and Cloudflare)
But this is all moot when it comes to XMPP, since XMPP clients don't check for SCTs, and there is no requirement for CAs to log certificates if they don't need to work in non-CT-enforcing clients. Some CAs (e.g. DigiCert) will sell you unlogged certificates - no need to compel them.
tldr: situation is good and improving with browsers, not so much with non-browser clients
> Could you prevent or monitor this kind of attack?
There are several indications which could be used to discover the attack from day 1:
> All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
> All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
This seems like the kind of thing domain registrars should do for you.
As an end user, how can I monitor if I’m being MITM’d? If everything I do is proxied through an attacker’s network, nothing I do online could be trusted to properly test if I were being proxied, right? I know apps can do cert pinning, but as an end user how can I validate that when connect to anything that it’s the real one and not a request from being middled?
One can get fingerprints of TLS endpoints ahead of time and/or from a known clean network then compare and contrast to what you see from the location you suspect might be MITM'd. I believe there are browser addons that display certificate fingerprints and alert if a certificate changes however one can click on the lock symbol and drill down to the same info.
A higher friction method would be to use a browser addon that would pin a certificate to a site. Not useful in the case of the Jabber client but useful if one was visiting Jabbers website to validate something. Some Jabber clients and servers have their own method described in another part of this thread [1] to protect against interception.
The openssl s_client method is likely the most versatile for testing from different locations, ports and applications.
Next time someone in Matrix discussion will ask "why we need Matrix when there is jabber/xmpp" show them this:
> All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.
The same attack could be mounted against a Matrix server tbh. But the fact Matrix is E2EE by default would mitigate it a bit, and if folks verified who they were talking to then the attack would be defeated.
> The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023
> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.
> Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible.
And what if the attacker tricked somehow the letsencrypt challenges?
Even after obtaining certificates allowing you to MITM, you have to actually find TM you can be a MI. In a targeted attack, this could be as easy and discreet as spoofing the target’s hotel WiFi. In a country without plentiful cross-border connections or a diverse backbone, a tap in the right IX could also work. (Famously, the NSA exploited the oligopoly the major consumer ISPs have in the US. Somewhat less famously, Roskomnadzor had to embark on a multi-year boiling of the frog to make Internet censorship in Russia even remotely workable, due to the diversity of the market and of the interconnects left over from the late nineties and early oughts, culminating in a requirement for every ISP in the country to patch MITM hardware into their network.)
But for this kind of thing to happen on every connection to the server being impersonated, you either have to bring a very big and publicly noisy hammer like a BGP hijack, or have the Internet upstream of the server cooperate. If the traceroute info in the post is to be trusted, in this particular case Linode and Hetzner themselves—or perhaps their datacenter operators—seem to be performing the intercept.
> If the traceroute info in the post is to be trusted, in this particular case Linode and Hetzner themselves—or perhaps their datacenter operators—seem to be performing the intercept.
The attacker effectively controlled the IP the domain was pointed to. If you have this, getting a cert issued from any CA is trivial - you've proved to them you control the domain in question.
We need to go back to snail mail or something, this whole .well-known thing just stinks. We added layers on top like CT and while sound ideas, they don't tend to do anything unless you are Google or FB.
> Yes, the fraudulent certificate is memorialized in Certificate Transparency (CT) databases (the indelible, publicly accessible records of ~all issued TLS certs). But, most website owners don't know what CT is, have no idea how to check it, and wouldn’t know what the results meant
Using DANE or so (RRs CAA/TLSA) can help in this case (when provider route traffic to specific port), but does not solve the problem completely.
If DNS record will be compromised, then hacker can setup "correct" TLSA/CAA record to trust his fake certificate.
As result, problem can be solved, if we will have reliable DNS subsystem, where is impossible to compromise DNS provider or infrastructure.
Public key pinning has been mostly abandoned on popular websites due to high risk of prolonged outages should someone make a mistake and mistakes do happen.
There are browser addons that replicate this behavior however and some Jabber clients/servers apparently have their own mechanism for this [1] but one can only hope everyone is using the combination of servers and clients that support this feature.
We are using self developed application layer security. It requires specialized client code and is terminated at a gateway that proxies requests to our normal applications.
I really, really hate the idea of this kind of eavesdropping.
let's encrypt is a public non-profit, who would definitely assist in finding out how domain control verification (DCV) happened - somehow that obvious line of pursuit was missing from the investigation. The mystery of the MAC 90:de:01:49:76:ae is interesting though, but CIA-custom-Made NIC might be a bit superfluous as an idea. Linode (acquired by Akamai) might use something in-house designed and OEM-manufactured, so I wouldn't treat it as strong evidence. Talk to Let's Encrypt - that will likely unveil what happened.
first the article jumped the gun when saying it's by Hetzner/Linode, as it as much could have been done by the carrier the data-center connects to and from what I know about German espionage/wiretapping by police law that would be far more likely
second pretty much _every_ country has laws which require carriers to help them wiretap in case of an investigation with appropriate court orders
thirdly even if it went through Hetzner/Linode instead of carriers it wasn't done "a move from them" but something they where legally binding ordered to silently tolerate
lastly if as unlikely as it seems it was not lawful interception of police or similar then they (or a carrier) would have been hacked, i.e. there is absolutely no chance that server providers will do such an attack on their own violation, especially Hetzner (they also don't have the legal means to get the necessary certificates)
That "lawful" interception allows certificate issuance to be a means of wiretap completely undermines any trust one should have in CAs. It seems that an alternative is greatly needed.
The fact that they were legally required to MITM their customers does not make them more trustworthy (in the sense of unlikely to do it in the future), just the opposite!
Of course that applies equally to any other cloud host (modulo jurisdiction games) but that does little to restore my interest in running my software on other peoples computers.
I’m wondering if mTLS (aka zero-trust) is used, would take prevent this kind of MITM attack? My understanding is that it should since the cert is self-signed to be used from both ends.
so is jabber.ru still secure now? what needs to be done on the user side to make it so? can one reissue/request new server certificate from the server and not from let's encrypt? thanks
If you have used OMEMO or OpenPGP to encrypt your messages, the content of your messages is secure, provided you have verified the public keys of your contacts. If you didn't verify the keys, you should do it now and check that no suspicious keys have been injected.
The way that certbot works with let's encrypt the only surprise is that this does not occur more often. We have monitoring on several certs used for TLS. We should probably add an alert if an A-record changes as well.
In fairness to them if it's a lawful intercept then they have no choice. Non compliance can shut down a business.
Your only alternatives in that case would be the underground style hosters sometimes marketed as bullet-proof hosting though they are just shady resellers with bold claims. They do not typically last long. There are a couple I know of in Amsterdam that are right down the street from The Hague that have been around for a while so I suspect they are just honeypots.
> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.
This seems to be somewhat jumping the gun I think.
Given the certs and the target assuming it's an lawful interception seems reasonable.
But there is nothing there which requires Hetzner or Lindoe complying or knowing about this.
Given the nature of the attacks you can do the interception on the carrier level, and carriers being forced to comply with lawful wiretapping is pretty much anywhere in the world pretty much standard and many laws are based around that approach. Much less so around approaches involving data centers.
according to the article, the added hop visible in traceroute to Linode machine is after two 10/8 hops, so it's likely an internal machine. more importantly, the affected VM has a strange gateway MAC address, which can only be controlled by the local network administrator, not an intermediate carrier.
Getting a valid cert for a domain you don't own is getting close to trivial. All of the mitigation strategies are defeated a number of ways, have been for years. I've commented on incidents like these for years on HN. I'm not some internet big wig, and any blog I write isn't going to trend, so this never gets traction. But anyone who understands the tech can find a half dozen ways to get a cert. If you really need to trust your connection, don't rely on internet PKI / public certs. It was secure enough for general purpose in the aughts, it's not anymore.
This would not be trivial and would require successful phishing. I've sent an email requesting to make this even harder to do. leaving out those details on purpose
It's really not that bold. Ask anyone who works for a CA, or an ISP, or a registrar, or a mail hoster, or a browser manufacturer, or DNS provider... If you just read up on how PKI works, on how certificates get issued, how the network protocols work, etc, it's all just right there. The holes are very well known and understood and not fixed because they're in the design. Every so often somebody comes up with another mitigation, which of course you have to opt into, and still doesn't solve all the holes.
It's like SS7, or SWIFT. It's not bold to claim that all public phone networks and bank transfers are insecure, it's just a fact. People who know what they are know they're vulnerable.
As an aside to the design issues, social engineering any of the thousands of organizations involved with issuing certs (not just CAs) is trivial. A bored teenager could get certs issued for most domains by using nothing more than a telephone or e-mail. But that's cheating, so I don't count it.
This is almost certainly related to some sort of Russian cybercrime investigation. If you ever read Krebs or peruse some of the seedier Russian forums, xmpp.ru will sound familiar to you, because it is. Not imputing anything to the operator, that's just the nature of operating an anonymous service.
https://ddanchev.blogspot.com/2021/03/exposing-currently-act...
https://ddanchev.blogspot.com/2019/07/profiling-currently-ac...
https://blog.talosintelligence.com/picking-apart-remcos/
https://flashpoint.io/wp-content/uploads/Plea-Agreement-USA-...
Really interesting writeup though. I guess it's a practical example of why everyone should get a CT monitoring service!
Drug gangs widely use XMPP as a secure communication channel in combination with Tor. Hydra (the darknet market) has been taken down by German police in 2022 after trying to extend their "service" to the EU; something similar might or might not have happened here.
[flagged]
27 replies →
Yes, I tend to believe that as well. My guess it's either related to Genesis Market or Qakbot takedowns.
[dead]
[flagged]
This criticism/argument was, in fact, made against domain validation (DV) when it was first introduced by earlier CAs. It's not an unreasonable criticism, but the economics of PKI mean that we would have much, much less encryption and authentication on the web today without DV.
DV represents a security trade-off, and Let's Encrypt took this and ran with it, with the net effect of much much much much less web traffic interception overall, albeit with known non-Let's-Encrypt-specific vulnerabilities to attackers who can manipulate infrastructure sufficiently.
As other people in this thread have noted, there are also other mechanisms that can help mitigate those vulnerabilities. Maybe we can come up with more over time!
what? nothing to do with Let's Encrypt, if someone gets to large-scale MITM your traffic then they can get anyone to issue a DV cert.
4 replies →
> This shows that Let's Encrypt security is a joke because now any large national ISP can use same MiTM to issue a certificate for any site hosted within a country. The SSL infrastructure is completely compromised.
the information available right now are too vague to come to a conclusion this bold.
Instead, I find it something like the following more plausible:
jabber.ru and xmpp.ru seem to use "exotic" DNS servers (at least as I checked right now).
https://uk.godaddy.com/whois/results.aspx?itc=dlp_domain_who...
All it then takes is an exploit there in the DNS server, or a badly set-up ACME DNS-01 there, in order for Let's Encrypt to grant an SSL certificate.
https://letsencrypt.org/docs/challenge-types/#dns-01-challen...
The moment you're able to write a (TXT) record for some domain name, you have proven to be eligible for getting an SSL certificate for that domain name.
6 replies →
I agree with most of the mitigation suggestions.
For high risk targets, consider layering an additional auth mechanism that doesn't rely on trusted CAs: Tor onion services, SSH, or Wireguard.
> All issued SSL/TLS certificates are subject to certificate transparency.
+1. crt.sh has RSS feeds for this.
> Limit validation methods and set exact account identifiers
Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?
> Monitor SSL/TLS certificate changes on all your services using external service
+1. High-risk targets should be aware of what certs are valid at any time, and be checking for those.
> Monitor MAC address of default gateway for changes
A more sophisticated attack could preserve the MAC address.
> "Channel binding" is a feature in XMPP which can detect a MiTM even if the interceptor present a valid certificate.
TIL.
>Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?
Author of ACME-CAA (RFC 8657) here. ACME-CAA can mitigate this because you can put a unique identifier for your ACME account in the CAA record, so it is not possible for an attacker to do this unless they can get your ACME account private key (or coerce the ACME service). This assumes you have DNSSEC-secured nameservers, of course, otherwise DNS requests can potentially also be intercepted when queried by the CA.
See RFC 8657 for a full list of security caveats. The RFC is designed to be more readable than most.
Blogpost by me with more background: https://www.devever.net/~hl/acme-caa-live
Small update: I've just written a blogpost with my thoughts on the incident. https://www.devever.net/~hl/xmpp-incident
I usually brood on blogposts for days before publishing them since I care a lot about getting things accurate, but this is a bit of the moment, so here goes. Always happy to get feedback by email or IRC: https://www.devever.net/~hl/contact
3 replies →
Wuuut, I missed they finally enabled it. Everyone who has infra MitM in their threat-model should look into and enable this.
Also, thank you for your work as well as popping in here.
Am I overstating it by seeing it as one of the most important milestones for Internet security in general over the past few years?
Do you have a take on DLT-based systems and if this is something that is (or could be) seriously discussed? It seems to me that the issues we have with PKI and certificate transparency could actually be mitigated very well if blockchain was seriously considered.
1 reply →
Interesting and I can see some of the benefits, particularly in preventing DNS controllers from going to other CAs and making new keys, but... this all seems like a weird run-around for the MITM part.
Couldn't you exchange public keys with LetsEncrypt (in the web UI) and encrypt the response so you can't be MITMed? Why is http even an issue?
4 replies →
Could you please elaborate the reasons to implement this outside of DANE (RFC-7671) framework?
13 replies →
Yep! This is one way of preventing this type of attack.
> All issued SSL/TLS certificates are subject to certificate transparency.
Until the nation state tells a CA within their borders to issue a cert without publishing the CT logs, along with a gag order. If they don't comply they go to jail. You think Billy Bob Sysadmin wants to go to jail to protect a russian jabber server?
The malicious certificate either won't have a SCT (which in itself would be highly suspicious and cause some clients to reject the certificate directly), or it will have one but it won't be recorded in the log.
In the latter case, the malicious certificate + any CT log issued after that certificate should have been included are evidence of the attack that is easily verifiable and would likely cause browsers to drop the (edit) CT log unless provided with a very plausible excuse that isn't "we complied with a court/gag order".
Also, this would require the police etc. to compel 2-3 different entities: The CA and two log operators (I think one of them could be the CA itself).
Using different logs than normal would stand out, so it'd typically be two specific log operators that would have to be compelled to hide this. The US may get lucky and have jurisdiction over all of them, other countries are much less likely to.
4 replies →
If the certificate isn't logged in CT, it's not supposed to be trusted. That's the whole point. If it is logged, then the site owners would notice (Cloudflare, for example, has a service that emails when certificates are issued for your domains).
Browsers will check if a certificate is in the transparency log, and alert the user if it isn't if I am not mistaken.
5 replies →
Just to be clear (with better details already given by other comments), addressing this is specifically part of the threat model for Certificate Transparency. You could argue that CT policies should be strengthened in various ways, but the threat model does intend to address detecting the compelled misissuance scenario.
CA being in one country and hosting in another greatly complicates this. Ideally the two countries shouldn't be part of Five Eyes or some such.
If they have physical access to your server, they'll just go for that.
With mild difficulty you can mirror the disk, net, and RAM of any "commodity" hardware.
[flagged]
How would you implement “strict checking” at scale and for free?
8 replies →
Perfectly pulling off an actual MitM attack and then forgetting to renew the certificate is certainly a very German thing :-)
I wonder if someone didn't "forget" on purpose, so that people learn about it.
Or, someone very diligently followed the orders - there was an order to set up a cert, but there was no requirement that it has to auto-renew :)
It was claimed to have been running for 6 months so it must have renewed certs at least once - LE certs are good for 90 days.
7 replies →
Any sanely designed covert transparent proxy software will automatically stop proxying when the certificate expires.
I wonder why this didn't.
No, the German thing would have been to have the printout of the telefaxed scan of the court order collecting dust because the scanner in the receiving department is broken...
How is that possible while Letsencrypt keeps on sending reminder emails?
You expect German institutions to accept electronic mail?
Being Germany, perhaps the paperwork expired and the new court order didn't arrive in time.
Archive [1]
Their theories are interesting and would explain how they obtained the certs if true. Perhaps the take-away here is to have multiple probes on multiple providers checking all of ones TLS fingerprints and alerting on unknown fingerprints and then checking the certificate transparency log or lack thereof.
[1] - https://archive.ph/C0jYJ [updated]
I've added "Could you prevent or monitor this kind of attack?" section.
There are several indications which could be used to discover the attack from day 1:
* All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
* Limit validation methods and set exact account identifier which could issue new certificates with Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (RFC 8657) to prevent certificate issue for your domain using other certificate authorities, ACME accounts or validation methods
* Monitor SSL/TLS certificate changes on all your services using external service
* Monitor MAC address of default gateway for changes
Nice addition. Me personally being the paranoid type, I don't trust the transparency log for monitoring but rather for writing up the root cause analysis. Reason being is that if someone can legally compel Hetzner and/or Linode and/or LetsEncrypt to do or not do something then the same entity can compel the certificate transparency site to ignore something. But you covered what I would do and that is to have multiple nodes doing active monitoring of TLS changes using an external service. That service being openssl s_client in my case.
In the case of Jabber it might be interesting to add some monitoring in the application that uses a cryptographic signed payload to list all the valid fingerprints and send an alert message to the server if something odd is happening like public key pinning but without a hard-fail. That list could be pre-loaded with new certificate fingerprints prior to being deployed. If the oddness is confirmed then perhaps add some way to tell the clients that certificate is likely forged. That way both server operator and client are aware that something evil this way comes.
[Edit] - Looks like based on the comment from MattJ100 that some Jabber servers and some clients can already do something like this.
9 replies →
I mentioned this in the operators channel just now, but also worth noting is that channel binding (e.g. SCRAM-*-PLUS in combination with RFC 9266) also mitigates this attack. Essentially the connecting client is able to detect the mismatch between the certificate or handshake the server thinks it is presenting, and the one the client actually sees.
ejabberd and Prosody support it, and a number of clients too.
3 replies →
DANE (with DNSSEC) may be an option to look at too, with public keys in DNS. though probably client support isn't universal (i have no (recent) experience using jabber). the CA was one weak point here.
checking for new public keys in certificates when keeping private keys between renewals would also help.
i suppose any storage and possibly ram should also be considered compromised? e.g. private keys stored on disk without encryption, but also by being read into memory. especially on the vm.
12 replies →
There's also a DNS CAA record that sets up a whitelist of authorities that can issue certificates for the domain.
7 replies →
Let's encrypt (and other CAs) should not allow validation using unsecure schemes like HTTP. Especially for sites that have valid certificate from other CA.
I was a bit curious about the MAC address since it isn't locally administered which indicates to me that someone likely deliberately chose this range of addresses.
https://www.google.com/search?q=%2290%253Ade%253A01%22+linod...
Googling for '"90:de:01" linode' and '"90:de:00" linode' indicates that addresses from these blocks have been assigned to other linode VMs recently. I sadly don't have a linode VM of my own to compare with right now but it would seem like the traffic has been routed to another VM on linode infrastructure
This is baseless speculation, but I'm assuming Jabber is being targeted as it's famously used on darknet markets for drug trades (or other illicit activity). Goes to show that you should never just trust "it's encrypted, bro". You need to PGP your messages at the very least. Is PGP crackable by quantum computers? Will there be hardening against those kinds of attacks in the future? Since, if the messages have been hoovered up in encrypted form, it's just a matter of time until they get decrypted. And this appears to be done for just about all web traffic they can get their hands on... see https://en.wikipedia.org/wiki/Utah_Data_Center
Jabber/XMPP has had e2e encryption for at least like 10-15 years. I used to use it with even my normie friends back when Facebook/Google Talk supported XMPP and you could use pidgin, kopete, etc.
Obviously securely exchanging keys with an anonymous drug dealer over the Internet is error-prone though...
This mitm may be as a part of Genesis Market takedown, but it's just out of the blue.
https://therecord.media/genesis-market-takedown-cybercrime
A second layer of encryption would help, but I don't recommend PGP in particular.
If you haven't heard, it has lots of problems and a lot of people recommend avoiding it (for example https://news.ycombinator.com/item?id=20455780)
"The PGP Problem" is generally misleading and is straight out wrong in some places. I ended up writing an article to save time:
* https://articles.59.ca/doku.php?id=pgpfan:tpp
PGP certainly has its problems, but isn't really special compared to other similar things. The big advantage that PGP has is that it is a stable and well known standard. There is a tendency to imply that it is insecure in some way, but no real evidence seems to exist to that effect.
1 reply →
Most clients also support OMEMO now
5 replies →
>End-to-end encrypted communications, such as OMEMO, OTR or PGP, are protected from the interception only if both parties have validated the encryption keys. The users are asked to check their accounts for new unauthorized OMEMO and PGP keys in their PEP storage, and change passwords.
The attacker would need to set up a separate MiTM for the particular E2EE scheme used. Some of the XMPP clients I have encountered will not let you use a particular cryptographic identity unless you have explicitly claimed to have verified it.
Still a good reminder. If you have not seen and dealt with a ridiculously long number (or equivalent) you have not achieved end to end.
There's one thing I can't understand in this story: if that's lawful interception, why Hetzner and Linode bothered to set up MitM interception with different LE certificate and key, rather than extract the TLS private key directly from the RAM and/or storage device of the VPS? Even if this is a physically dedicated server, they can extract the private key from the RAM by dumping the RAM contents after unscheduled reboot. Extraction of the private key isn't visible in CT logs, much more stealthier, practically undetectable.
Because it was easier, most likely.
There's also a possibility that one would be a "search" and the other would be an "interception" with different levels of approvals requested, but I don't know what the current legal situation in Germany is right now.
Likely because it's 'more illegal'. I'd bet they are not allowed to hack into the server if it's not directly involved in the cybercriminal activity.
On a physical server, couldn't you just hotplug a PCIe card in there and DMA out any data you are interested in? Something like a network card with firmware specifically for the purpose should do it. It sounds so much a standard thing for law enforcement that I imagine such equipment should be available off the shelf?
A user reported this on Reddit 3 days ago: https://www.reddit.com/r/hetzner/comments/17ankoh/does_hetzn...
I wonder if there is a legal way to demand Hetzner/Linode comments on this situation. Likely, the entity behind the interception is some government agency or police.
For Hetzner as German Company: If it is legal interception no. You can’t demand an answer, without going to a lawyer first.
On the other hand, if it is not lawful interception, I doubt Hetzner would allow it. Because that’s also against law.
there isn't even any guarantee that the wiretapping was done through them instead of e.g. the carriers which pretty much in any country have since decades been forced to help with lawful wiretapping...
Carriers meaning the interconnect providers eg Level3, Cogent etc? How would this intercept be implemented in practice? Surely it'd be much easier to add a node as close as possible to the origin host, i.e. within the Hetzner network, rather than redirecting traffic from the outside with some sort of BGP hijack?
2 replies →
I think the observed TTL 64 means the interceptor is on the same segment? (of course unless they have set it to e.g. 66 at the interceptor that is 2 hops away, but I guess if they were to mangle TTL, they would set it to the original value to avoid detection)
[dead]
This really wants me to re-evaluate all the times I have gracefuly TOFU'd while SSHing into a machine or even logging onto a website.
Consider including SSH known hosts in your cfgmgmt.
I have all my machines' public keys, and all major git forge's SSH keys in my git-managed cfgmgmt repo.
For NixOS users: https://search.nixos.org/options?channel=23.05&show=programs...
one issue is that most hosting providers don't display ssh host keys anywhere in the control panels so unless you have a... lot of fun (and spend time) with the virtual console... (or maybe a custom cloud-init script?) you're SOL for the very very first connection
2 replies →
Given that the attack subverted the TLS certificates, I'd say key-based TOFU is still a better alternative.
Of course, checking the server key is better than not doing so, but any knee-jerk response to TOFU will probably make your security worse.
What does TOFU mean in this context? I haven’t heard that word related to tech before OpenTofu just recently!
Trust On First Use. No relation to OpenTofu where Tofu is a word play in TerraForm.
https://en.wikipedia.org/wiki/Trust_on_first_use
Trust On First Use
At DEFCON, fifteen years ago, MITM'ing TOFU ssh connections was what you did when you were bored in between talks. If you wanted to capture a new first connection, just keep dropping connections until they try a different host or device.
I don't really get why people use ssh at all if they aren't using certs. Might as well use telnet.
An unknow key when connecting to a known server will raise all kinds of alarms in my head.
I've always thought a solution like LetsEncrypt would be an ideal backdoor for LEAs wanting to circumvent SSL/TLS.
The same thing applies to VPN services. Why bother trying to develop tools etc that can decrypt traffic when you can just have people send their traffic to you?
Why are your concerns specific to LetsEncrypt? They're just another CA, no?
Wouldn't be surprised if this happened in Russia; but not in Europe. Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability - doesn't this mean that large national ISPs can easily issue ceritificates for any site hosted within a country?
> Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability
They can’t use HTTPS if you don’t have a certificate yet
This doesn't mean you should use unsecure methods instead and issue certificates to anyone capable to do MitM.
13 replies →
Police raiding / intercepting / modifying servers does happen here too. Last big case: https://www.bleepingcomputer.com/news/security/encrochat-tak...
Could this be a blue pill attack? A vulnerability in the xmpp server exploited to inject a rootkit, which then hides itself inside the kernel?
Or creates network/pid namespaces and puts you in them, while leaving the mitm server in the original one?
If so, the mitm could be on the same host, and wouldn't need the cooperation of the hosting provider.
I'm not sure how to check for either of these without restarting (which the admin does not seem to want to do, as it is a live service).
https://en.wikipedia.org/wiki/Blue_Pill_(software)
If this had happened, the attacker would have likely stolen the servers TLS certificate and keys
Whereas this attack generated new keys (and was detected!), suggesting the attacker didn't compromise the server itself.
No because traffic redirection occured in provider's infrastructure, not on a server.
Let's say you are blue-pilled and what you see as your eth0 interface is actually a virtual interface controlled by the rootkit.
In that case, where the redirection happened is no longer something you would be able to tell, right?
>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023
Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?
It's fairly common for people to obtain multiple certificates for different machines or services, so they can be selectively revoked and they don't have to share keys across machines.
More use-cases:
- You might obtain a new certificate, but deploy it gradually, so you want the old one to remain valid while you do that.
- One certificate may cover different sets of domain names. If you have a certificate for "example.com, foo.example.com" and then request a certificate for only "foo.example.com", should the earlier one be revoked? (leaving "example.com" without a certificate).
> Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt?
it commonly used in a "normal" way all the time
- e.g. when there are multiple data-center for the same domain (e.g. using geo-location based routing) it's a good practice to give them different certs so that if you need to revoke one the operation in other regions is unaffected
- or when rolling over from on cert to another
- or when moving certs into hardware security keys/module (HSK) you preferably do have one per HSK (so that if e.g. hardware breaks and gets replaced you can just revoce the cert for the affected HSK module not all of them), you also normaly do not keep backups to make sure it can't be leaked at all (as long as the HSK isn't hacked which is normally quite hard)
- or losing access to a cert (e.g. in the case above a HSK breaks)
Lastly the whole CA system is in the end designed to provide good security for the industry while having the backdoor of issuing certs the legal organs to allow the police some degree of wiretapping (oversimplified, it's slightly more complex then that).
You should always have more than a single certificate for your domain honestly.
Cloudflare for example, tries to optimize certificate delivery (and have backup certificates available for you just in case a CA needs to revoke theirs).
Also, on distributed systems its less safe to share private keys between the various frontends.
This is actually a great suggestion and ACME providers should provide it as an opt-in feature via CAA record. Not even the provider having access to system memory could issue a mitm cert without you noticing.
The provider having access to system memory can copy the private key and use your original key+cert for MITM, unless you are using some fancy HSM.
provisioning a 2nd machine into your webserver cluster before activating it?
You could sync certificates across hosts for this purpose, though. The advantage of multiple certificates is being able to revoke a subset of certificates if you can determine only a subset of your hosts have been compromised.
2 replies →
> Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?
First, you want to have to have some leeway so you don't need to rotate certs at exact second the old one expires
Second, you might want to have cert-per-server rather than cert-per-domain, as that's frankly easier to implement vs having common store for certs+key
Are there any services that monitor the certificate transparency logs and send an email when a new certificate is issued for your domain? That would alert you to this kind of MITM
As mentioned there are services for doing this (I've been using for example the free service from Facebook [1])
Since the certificates are nowadays renewing quite often, it's very easy to just end up ignoring these notifications. To really get the benefits, I think you should somehow combine these with logs from the legit cert renewals and only alert when something strange pops up.
[1] https://developers.facebook.com/tools/ct/search/
Indeed, a CT monitor which sends alerts about legitimate certificates is almost useless due to noise. My service, Cert Spotter, provides an API endpoint[1] which you can upload your CSRs to, so you don't get alerted about certificates using the same key as the CSR. The open source version of Cert Spotter can invoke a script[2] when a certificate is discovered, and the script can cross reference against a list of legitimate certs.
[1] https://sslmate.com/help/reference/certspotter_authorization...
[2] https://github.com/SSLMate/certspotter/blob/master/man/certs...
2 replies →
Yes. Several. They're called CT "Monitors".
There are but this assumes that services like the transparency log servers and LetsEncrypt are not being legally compelled to do or not do something.
If the CA doesn't log a certificate, the certificate won't work in CT-enforcing clients (Chrome, Safari, Edge) because it lacks receipts (SCTs) from CT logs.
If a log fails to publish a certificate despite issuing a receipt for it, Chrome's SCT auditing infrastructure can detect that, as it did recently with the Nessie 2023 log: https://groups.google.com/a/chromium.org/g/ct-policy/c/5x1A6... (this was also detected by monitors operated by myself and Cloudflare)
But this is all moot when it comes to XMPP, since XMPP clients don't check for SCTs, and there is no requirement for CAs to log certificates if they don't need to work in non-CT-enforcing clients. Some CAs (e.g. DigiCert) will sell you unlogged certificates - no need to compel them.
tldr: situation is good and improving with browsers, not so much with non-browser clients
4 replies →
> Could you prevent or monitor this kind of attack? There are several indications which could be used to discover the attack from day 1:
> All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
> All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
This seems like the kind of thing domain registrars should do for you.
2 replies →
Cloudflare does that
Cloudflare is a MITM you voluntarily setup yourself, innit?
1 reply →
Kerberos right?
As an end user, how can I monitor if I’m being MITM’d? If everything I do is proxied through an attacker’s network, nothing I do online could be trusted to properly test if I were being proxied, right? I know apps can do cert pinning, but as an end user how can I validate that when connect to anything that it’s the real one and not a request from being middled?
One can get fingerprints of TLS endpoints ahead of time and/or from a known clean network then compare and contrast to what you see from the location you suspect might be MITM'd. I believe there are browser addons that display certificate fingerprints and alert if a certificate changes however one can click on the lock symbol and drill down to the same info.
A higher friction method would be to use a browser addon that would pin a certificate to a site. Not useful in the case of the Jabber client but useful if one was visiting Jabbers website to validate something. Some Jabber clients and servers have their own method described in another part of this thread [1] to protect against interception.
The openssl s_client method is likely the most versatile for testing from different locations, ports and applications.
[1] - https://news.ycombinator.com/item?id=37956911
https://github.com/matteocorti/check_ssl_cert
@dang: .org.ru should probably be treated as TLD
Next time someone in Matrix discussion will ask "why we need Matrix when there is jabber/xmpp" show them this:
> All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.
The same attack could be mounted against a Matrix server tbh. But the fact Matrix is E2EE by default would mitigate it a bit, and if folks verified who they were talking to then the attack would be defeated.
Matrix clients typically complain very loudly when keys change or unverified sessions get added to your chats.
Id's say it would mitigate it A LOT. Though I have to agree that there is still a lot of room for improvement.
I personally prefer the approach taken by briar: your public key is your address. Pretty cut and dry, not much room for shenanigans.
> The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023
> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.
> Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible.
And what if the attacker tricked somehow the letsencrypt challenges?
Or this is supposed to be impossible?
Even after obtaining certificates allowing you to MITM, you have to actually find TM you can be a MI. In a targeted attack, this could be as easy and discreet as spoofing the target’s hotel WiFi. In a country without plentiful cross-border connections or a diverse backbone, a tap in the right IX could also work. (Famously, the NSA exploited the oligopoly the major consumer ISPs have in the US. Somewhat less famously, Roskomnadzor had to embark on a multi-year boiling of the frog to make Internet censorship in Russia even remotely workable, due to the diversity of the market and of the interconnects left over from the late nineties and early oughts, culminating in a requirement for every ISP in the country to patch MITM hardware into their network.)
But for this kind of thing to happen on every connection to the server being impersonated, you either have to bring a very big and publicly noisy hammer like a BGP hijack, or have the Internet upstream of the server cooperate. If the traceroute info in the post is to be trusted, in this particular case Linode and Hetzner themselves—or perhaps their datacenter operators—seem to be performing the intercept.
> If the traceroute info in the post is to be trusted, in this particular case Linode and Hetzner themselves—or perhaps their datacenter operators—seem to be performing the intercept.
Ok, now I get it. Thanks.
The attacker effectively controlled the IP the domain was pointed to. If you have this, getting a cert issued from any CA is trivial - you've proved to them you control the domain in question.
As mentioned elsewhere in the thread, RFC 8657 can prevent this.
https://news.ycombinator.com/item?id=37958831
They don't need to, they controlled the domain IP and trivially got the certificates. This is not a novel technique, see this Twitter thread:
https://twitter.com/billmarczak/status/1710348549794185279
We need to go back to snail mail or something, this whole .well-known thing just stinks. We added layers on top like CT and while sound ideas, they don't tend to do anything unless you are Google or FB.
> Yes, the fraudulent certificate is memorialized in Certificate Transparency (CT) databases (the indelible, publicly accessible records of ~all issued TLS certs). But, most website owners don't know what CT is, have no idea how to check it, and wouldn’t know what the results meant
CAA account binding is the response to this.
If that were the case, they wouldn't have seen weird connectivity issues/behaviors.
why not? They discover the issue only because the attacker failed to renew the certificates.
1 reply →
Using DANE or so (RRs CAA/TLSA) can help in this case (when provider route traffic to specific port), but does not solve the problem completely. If DNS record will be compromised, then hacker can setup "correct" TLSA/CAA record to trust his fake certificate. As result, problem can be solved, if we will have reliable DNS subsystem, where is impossible to compromise DNS provider or infrastructure.
I think, emerDNS can be used in such critical application: https://emercoin.com/en/documentation/blockchain-services/em...
And this is why, kids, you need to enforce certificate pinning on your critical infrastructure!
Public key pinning has been mostly abandoned on popular websites due to high risk of prolonged outages should someone make a mistake and mistakes do happen.
There are browser addons that replicate this behavior however and some Jabber clients/servers apparently have their own mechanism for this [1] but one can only hope everyone is using the combination of servers and clients that support this feature.
[1] - https://news.ycombinator.com/item?id=37956911
We are using self developed application layer security. It requires specialized client code and is terminated at a gateway that proxies requests to our normal applications.
I really, really hate the idea of this kind of eavesdropping.
Interesting reddit thread ending here: https://www.reddit.com/r/hetzner/comments/17ccp3i/hetzner_do...
let's encrypt is a public non-profit, who would definitely assist in finding out how domain control verification (DCV) happened - somehow that obvious line of pursuit was missing from the investigation. The mystery of the MAC 90:de:01:49:76:ae is interesting though, but CIA-custom-Made NIC might be a bit superfluous as an idea. Linode (acquired by Akamai) might use something in-house designed and OEM-manufactured, so I wouldn't treat it as strong evidence. Talk to Let's Encrypt - that will likely unveil what happened.
Interestingly, search "90:de:01" in this page: https://www.cyberciti.biz/faq/howto-linux-configuring-defaul...
^^ looks like another victim VM of the alleged mysterious interceptor :-)
What a nefarious move by Hetzner and Linode. How to trust them after this?
first the article jumped the gun when saying it's by Hetzner/Linode, as it as much could have been done by the carrier the data-center connects to and from what I know about German espionage/wiretapping by police law that would be far more likely
second pretty much _every_ country has laws which require carriers to help them wiretap in case of an investigation with appropriate court orders
thirdly even if it went through Hetzner/Linode instead of carriers it wasn't done "a move from them" but something they where legally binding ordered to silently tolerate
lastly if as unlikely as it seems it was not lawful interception of police or similar then they (or a carrier) would have been hacked, i.e. there is absolutely no chance that server providers will do such an attack on their own violation, especially Hetzner (they also don't have the legal means to get the necessary certificates)
That "lawful" interception allows certificate issuance to be a means of wiretap completely undermines any trust one should have in CAs. It seems that an alternative is greatly needed.
The fact that they were legally required to MITM their customers does not make them more trustworthy (in the sense of unlikely to do it in the future), just the opposite! Of course that applies equally to any other cloud host (modulo jurisdiction games) but that does little to restore my interest in running my software on other peoples computers.
3 replies →
If it's a "lawful intercept" then they most likely have no choice.
I’m wondering if mTLS (aka zero-trust) is used, would take prevent this kind of MITM attack? My understanding is that it should since the cert is self-signed to be used from both ends.
A reminder e2ee is a must.
so is jabber.ru still secure now? what needs to be done on the user side to make it so? can one reissue/request new server certificate from the server and not from let's encrypt? thanks
If you have used OMEMO or OpenPGP to encrypt your messages, the content of your messages is secure, provided you have verified the public keys of your contacts. If you didn't verify the keys, you should do it now and check that no suspicious keys have been injected.
The way that certbot works with let's encrypt the only surprise is that this does not occur more often. We have monitoring on several certs used for TLS. We should probably add an alert if an A-record changes as well.
I assume the root cause is DNS tricks.
> I assume the root cause is DNS tricks.
Maybe read the article
well i must admit monitoring certificate issuance with LetsEncrypt is quite boring even if you HAVE alerts set up (I do)
so… not surprised. Still cool. What a time to be alive
So, if I rent server in those services, it all potentially can be compromised and there is no way to make sure things are secure?..
? did you not read the article?
I briefly read the article, maybe you can explain what is your point?
10 replies →
http://archive.today/0R7Or
This is a deal breaker for me. Which is a shame because Hetzner has such great pricing. Oh well.
In fairness to them if it's a lawful intercept then they have no choice. Non compliance can shut down a business.
Your only alternatives in that case would be the underground style hosters sometimes marketed as bullet-proof hosting though they are just shady resellers with bold claims. They do not typically last long. There are a couple I know of in Amsterdam that are right down the street from The Hague that have been around for a while so I suspect they are just honeypots.
It's likely legally enforced, and any other provider would be subject to the same.
Fair enough
> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.
This seems to be somewhat jumping the gun I think.
Given the certs and the target assuming it's an lawful interception seems reasonable.
But there is nothing there which requires Hetzner or Lindoe complying or knowing about this.
Given the nature of the attacks you can do the interception on the carrier level, and carriers being forced to comply with lawful wiretapping is pretty much anywhere in the world pretty much standard and many laws are based around that approach. Much less so around approaches involving data centers.
according to the article, the added hop visible in traceroute to Linode machine is after two 10/8 hops, so it's likely an internal machine. more importantly, the affected VM has a strange gateway MAC address, which can only be controlled by the local network administrator, not an intermediate carrier.
Getting a valid cert for a domain you don't own is getting close to trivial. All of the mitigation strategies are defeated a number of ways, have been for years. I've commented on incidents like these for years on HN. I'm not some internet big wig, and any blog I write isn't going to trend, so this never gets traction. But anyone who understands the tech can find a half dozen ways to get a cert. If you really need to trust your connection, don't rely on internet PKI / public certs. It was secure enough for general purpose in the aughts, it's not anymore.
What are these trival steps to get a valid certificate for news.ycombinator.com?
This would not be trivial and would require successful phishing. I've sent an email requesting to make this even harder to do. leaving out those details on purpose
this is left as an exercise for the reader
Bold claim lacking substantiation. Care to share? Maybe it will get traction this time.
It's really not that bold. Ask anyone who works for a CA, or an ISP, or a registrar, or a mail hoster, or a browser manufacturer, or DNS provider... If you just read up on how PKI works, on how certificates get issued, how the network protocols work, etc, it's all just right there. The holes are very well known and understood and not fixed because they're in the design. Every so often somebody comes up with another mitigation, which of course you have to opt into, and still doesn't solve all the holes.
It's like SS7, or SWIFT. It's not bold to claim that all public phone networks and bank transfers are insecure, it's just a fact. People who know what they are know they're vulnerable.
As an aside to the design issues, social engineering any of the thousands of organizations involved with issuing certs (not just CAs) is trivial. A bored teenager could get certs issued for most domains by using nothing more than a telephone or e-mail. But that's cheating, so I don't count it.
1 reply →
If you are your hoster and can reroute requests, you can get all kinds of valid SSL certs for any domains that you host.
Just grab these IP packets when CA comes to validate that you own that domain. Perhaps EV could solve that to some extent but it is never mandated.
Even if you tried to put any stuff into WHOIS to mitigate this, your hoster can serve any bullshit on this channel too.
It does look very bad and SSH approach to certificates is just infinitely better. If Jabber used SSL keys instead, they will be alerted immediately.
Come to think of it, your hoster can also find ways to steal keys directly from hardware, though.
2 replies →