Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :)
We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent). If so, we append the ephemeral root CA to the list of actual CA certificates; if not, we let the kernel handle the file open like usual. This way, none of the other programs are affected, so only the program you start with `subtrace run` sees and trusts the ephemeral root CA.
After we get the program to trust the ephemeral root CA, we can proxy outgoing TLS connections through Subtrace transparently but also read the cleartext bytes.
It won't work with programs that defensively validate the cert chain but those are rare.
It won't work with programs that embed their own root cert store, which is also rare but I would guess less rare than the previous one. The usual reason to do this is to minimize OS deps, and in the case of Docker containers to save on container image size by only including the roots you care about.
But yes for the vast majority of programs it should work fine.
Yep, certificate pinning is the one scenario Subtrace can't handle in my experience, but thankfully, it's fairly rare like you said. And IMO there is no general solution to the problem [1], but it's one of those very interesting problems to daydream thinking about when you're stuck in traffic or whatever :)
We still try our best by handling as much of the long tail of environments with some library/framework specific workaround (e.g. Deno bundles all TLS certs in its binary so we set the DENO_CERT env var when applicable).
Is this a different method from the httptap [1] that was on hackernews a few weeks ago? Somebody in that post seemed to say that it also generates CA certificates on the fly.
httptap is really cool! Their technique is different (they do a filesystem mount instead of intercepting syscalls like Subtrace does) but both tools effectively reach the same goal using different routes.
Up front, this is not my area of expertise. You would still not want to run this on the same server as your containers since someone could inject their own cert? I think it allows someone to decrypt traffic that isn't just proxied to Subtrace?
Edit: I've reviewed the docs and it looks like you do run it on the same server. For clarity, I've used Sentry before.
Subtrace proxies the program's connection using a regular TLS connection to the upstream server. For example, if you do `subtrace run -- curl https://example.com`, curl thinks it's talking to example.com over TLS, but it's really talking to Subtrace locally. Since we injected the ephemeral root CA into the system store, curl will trust the valid TLS certificate that Subtrace presents for example.com. From within the same server, Subtrace will handle the actual TLS connection to upstream example.com. That upstream connection is undecipherable to outsiders.
Everything is exactly as secure as before Subtrace. In other words, using Subtrace doesn't make the NSA's job any easier ;)
This is a great hack. It won’t work 100% of the time but it’ll be close, and damn it’s just so clean. Proxies and intercepts all the way down… I love this.
Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :)
We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent). If so, we append the ephemeral root CA to the list of actual CA certificates; if not, we let the kernel handle the file open like usual. This way, none of the other programs are affected, so only the program you start with `subtrace run` sees and trusts the ephemeral root CA.
After we get the program to trust the ephemeral root CA, we can proxy outgoing TLS connections through Subtrace transparently but also read the cleartext bytes.
All of this is fully automatic, of course.
This will not work with HPKP but hopefully nothing is using that any more. ( https://en.m.wikipedia.org/wiki/HTTP_Public_Key_Pinning )
It won't work with programs that defensively validate the cert chain but those are rare.
It won't work with programs that embed their own root cert store, which is also rare but I would guess less rare than the previous one. The usual reason to do this is to minimize OS deps, and in the case of Docker containers to save on container image size by only including the roots you care about.
But yes for the vast majority of programs it should work fine.
Yep, certificate pinning is the one scenario Subtrace can't handle in my experience, but thankfully, it's fairly rare like you said. And IMO there is no general solution to the problem [1], but it's one of those very interesting problems to daydream thinking about when you're stuck in traffic or whatever :)
We still try our best by handling as much of the long tail of environments with some library/framework specific workaround (e.g. Deno bundles all TLS certs in its binary so we set the DENO_CERT env var when applicable).
[1] https://news.ycombinator.com/item?id=42923998
4 replies →
Is this a different method from the httptap [1] that was on hackernews a few weeks ago? Somebody in that post seemed to say that it also generates CA certificates on the fly.
[1] https://news.ycombinator.com/item?id=42919909
httptap is really cool! Their technique is different (they do a filesystem mount instead of intercepting syscalls like Subtrace does) but both tools effectively reach the same goal using different routes.
Up front, this is not my area of expertise. You would still not want to run this on the same server as your containers since someone could inject their own cert? I think it allows someone to decrypt traffic that isn't just proxied to Subtrace?
Edit: I've reviewed the docs and it looks like you do run it on the same server. For clarity, I've used Sentry before.
Subtrace proxies the program's connection using a regular TLS connection to the upstream server. For example, if you do `subtrace run -- curl https://example.com`, curl thinks it's talking to example.com over TLS, but it's really talking to Subtrace locally. Since we injected the ephemeral root CA into the system store, curl will trust the valid TLS certificate that Subtrace presents for example.com. From within the same server, Subtrace will handle the actual TLS connection to upstream example.com. That upstream connection is undecipherable to outsiders.
Everything is exactly as secure as before Subtrace. In other words, using Subtrace doesn't make the NSA's job any easier ;)
1 reply →
I can't decide if I'm horrified or amazed by this :)
as with anything too clever and undocumented, you will be amazed until you waste a few weeks debugging an issue caused by it.
Wow. I think more network inspection tools should adopt an approach like this. Great job.
This is a great hack. It won’t work 100% of the time but it’ll be close, and damn it’s just so clean. Proxies and intercepts all the way down… I love this.