Comment by LeoPanthera
2 years ago
The summary of the attack from https://notes.valdikss.org.ru/jabber.ru-mitm/ is very interesting:
* The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023
* The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since 18 Apr 2023, affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223)
* The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner)
* The MiTM attack stopped shortly after we begun our investigation and network tests on 18 Oct 2023, along with tickets to Hetzner and Linode support team, however passive wiretapping (additional routing hop) is still in place at least on a single Linode server
* Neither servers appear to be hacked
* Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses
---
Neither that page, nor the page linked from here, mention certificate pinning, maybe because XMPP doesn't support it (I don't know), but if it did, wouldn't that have prevented this kind of attack?
XMPP supports channel binding, which is not mentioned in this post but would have prevented this attack. Unfortunately jabber.ru is running server software from 2016, and that old version doesn't support it.
Pinning is problematic these days, because certificates are short-lived and renewed frequently. Users would be constantly asked to accept new certificates on a monthly basis, and they wouldn't think twice about clicking 'Accept' on an attacker's certificate then.
Channel binding works regardless of the certificate, and ensures the TLS stream is terminated by the entity you exepect.
The focus on (lack of) CT/SCT support in XMPP clients puzzles me, because it would not have detected this attack at all, which used valid CT-logged certs from Let's Encrypt. SCT verification would help detect theoretical issuance from another CA if they had a strict CAA record in place (they did not). But even then, channel binding is more privacy-friendly and does not depend on a third party.
Pinnig is based on the keypair, not the cert. You can renew and not break pinning, right?
Also you can phase in a new cert with pinning.
Yes, you can pin the public key instead, which is generally more helpful. But most ACME clients (including the "official" certbot) default to rotating the key too. That can be disabled, but it's a problematic default for this use case which means clients can't just enable pinning.
2 replies →
If you care enough to pin, purchasing actual certificates isn't that hard is it?
> * The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner)
This is gold.
The absolute lack of giving a shit is one of your major clues this was a lawful intercept scenario.
Someone was forced to do it, but they didn’t personally agree with it so they eventually made a “mistake” to tip off the target?
There is the plain incompetence explanation: the hosting provider gave control of the operation to the government entity. The underpaid and indifferent government employee did the best they could with their level motivation and skill level.
I mean, I’ve seen the auto renew fail a lot with the certbot. They definitely should have checked it in the renew period to make sure it was working, but I feel for them
>affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223)
Why did they only target the STARTTLS port? On a related note, I would never use the STARTTLS port (opportunistic encryption) if I knew that the server had a regular TLS port...
>I would never use the STARTTLS port (opportunistic encryption) if I knew that the server had a regular TLS port...
That is what XMPP clients tend to do...
These days XMPP servers tend to default to requiring TLS on both 5222 and 5223 (Let's Encrypt has changed everything). Prosody does this for example. It doesn't even support port 5223 by default anymore. Port 5223 was never an official port assignment.
So it is very possible that the MiTM was only done on port 5222 because that was the only port that clients were using.
Easier to conceal the attack.
The MiTM attacker can pass through a command stream without STARTTLS. If they intercepted 5223 they would have to do their own client-side TLS handshake with the attacked server, which would look really obvious to anybody doing TLS fingerprinting on the server: all of a sudden, 100% of their clients have the exact same TLS fingerprint.
Stop outsourcing your PKI to ICANN, folks. Domains are not public keys.
They were doing their own TLS handshake - that's how the attack was discovered (the attacker presented a different certicate, which eventually expired, presumably due to negligence). They were decrypting and re-encrypting.
3 replies →
It is pretty simple. 5223 port is called "legacy SSL". So, because it is legacy, clients would, by default, use 5222 + starttls. I.e. majority of clients connect with 5222.
Also, I've never understood why they've moved 5223 (regular TLS) into deprecated. It was pretty useful to enable SSL on AWS ELB on this port. Which is not possible with 5222, because it does XML stuff before switching to TLS.
I would speculate it is to not keep 2 ports open or something, was a reason to move it to deprecated. + you can do some cleartext communication before switching to TLS (not like it is that useful).
>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt
This implies that the attacker was able to validate domain ownership of jabber.ru and xmpp.ru by either a) being able to respond to the ACME protocol on domain(s) hosted http OR b) being able to write to a DNS TXT file for the domain(s). This implies that the victim already lost control of their process server and/or their nameserver.
No. Most probably it merely implies someone upstream of them (presumably their hosting provider under compulsion from German law enforcement) intercepted and spoofed their unencrypted, unauthenticated ACME HTTP traffic.
Same can happen to you.
The same way they are intercepting the jabber port they can intercept any other port for the HTTP ACME challenge. No need to get involved at the name server level.
1 reply →
Okay, but the part I don't understand is why do it this way? If the hosting provider is a threat, they don't need new certs. They can just read your private keys off your disk and MITM with that.
4 replies →
On April 3rd the IP address for ns1.jabber.ru was changed [1]. Two weeks later the first LetsEncrypt certificate was issued. Could be related?
1. https://dns.coffee/nameservers/ns1.jabber.ru
How would you do certificate pinning if you don't control the clients?
My understanding is that certificate pinning is only possible if you control the clients, in which case you can embed which certificates are allowed directly in the client and bypass the whole web PKI.
In a situation with general-purpose clients connecting, how would they know which certificates are meant to be allowed? That's what the web PKI is used for.
Of course, if you do provide your own clients, this just moves the problem further up the chain - in this case the place where customers would download the custom client software would be compromised and a malicious client served instead.
> How would you do certificate pinning if you don't control the clients?
Well you cannot. If you were paranoid, you would perhaps supply a hash through some out-of-band mechanism, which would require manually updating for each new cert.
Obviously most people wouldn't ever want to do that.
Isn't this what those "key hash pictures" in WhatsApp/Signal are solving?
XMPP clients could implement such a mechanism, and if any certificate/domain along the path changes, the users in a conversation would be notified.
These are usually to validate the keys used in end-to-end encryption. Both parties must confirm that they see the same details, which confirms that the same keys are being used on both ends.
TLS client certificates.
Use them.
Stop using domains. Stop.
And how do you distribute those to anonymous users?
> Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses
This suggests a compromise of Hetzner and Linode network management.
> This suggests a compromise of Hetzner and Linode network management.
No it doesn't. It suggests that they both complied with a request from law enforcement.
So… compromised.
2 replies →
More likely the carrier upstream of them.
No, that wouldn't insert a hop between the server and the provider's network.
Is it given that this attack was done from within Hetzner?
As I understand the techniques applied; this could have also been done at another place on the network route to the targeted server.
Ie. by the telecommunications company delivering traffic into Hetzner.
Tele communication companies have internal “police groups” (where I am from - I expect it to be the same in Germany) that does nothing but service wiretapping requests from the police. Telcos are required by law to do this.
Them expanding into mtm https wiretapping is a new for me but maybe to be expected …