Comment by JohnLeitch
21 hours ago
The reliance on LLMs is unfortunate. I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark. The Wireshark dissectors are quite mature, SSH is covered fairly well.
I'm anti-LLM in most cases, but:
> I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark.
For some people who are used to using Wireshark and who know what to look for, probably yes. For the vast majority of even technical people, probably not.
In my case, I did a packet capture of a single keystroke using tcpdump and imported it into Wireshark and I get just over 200 'Client: encrypted packet' and 'Server: encrypted packet' entries. Nothing useful there at all. If I tcpdump the entire SSH connection setup from scratch I get just as much useful information - nothing - but, oddly, fewer packets than my one keystroke triggered.
So yeah, I dislike LLMs entirely and dislike the reliance on LLMs that we see today, but in this case the author learned a lot of interesting stuff and shared it with us, whereas without LLMs he might have just shrugged and moved on.
And thats a huge downside when people howl about "Encryption everywhere! ".
Try debugging that shit. Thats right, debugging interfaces aren't safe, by some wellakshually security goon.
You want a real fun one to debug, is a SAML login to a webapp, with internal Oauth passthrough between multiple servers. Sure, I can decrypt client-server stuff with tools, but server-server is damn near impossible. The tools that work break SSL, and invalidate validation of the ssl.
Yes, Esri products suck. Bad.
This really does not need to be that hard. For TLS, many tools support setting the SSLKEYLOGFILE environment variable to log the session keys used in connections. Wireshark can import those to decrypt everything. [1]
Unfortunately, nothing exists for SSH (yet?). [2]
I do agree that if you design a protocol that enforces encryption, you should include some debugging interface. It is much more straightforward to do this by logging the session secrets on the endpoints rather than trying to break it through a man-in-the-middle, the main thing the protocol is protecting you against.
[1]: https://wiki.wireshark.org/TLS
[2]: https://gitlab.com/wireshark/wireshark/-/issues/16054
I used to share that opinion but after decades in industrial automation I find myself coming down much more on the "yeah, encryption everywhere" because while many vendors do not provide good tools for debugging, that's really the problem, and we've been covering for them by being able to snoop the traffic.
Having to MITM a connection to snoop it is annoying, but the alternative appears to be still using unencrypted protocols from the 1970s within the limitations of a 6502 to operate life-safety equipment.
2 replies →
It seems like a leap to suggest we shouldn't have widely deployed encryption...rather than just fix the debugging tools.
Particularly in today's political climate, encryption has only become more necessary.
Sounds like blaming a tool on a problem it did not cause. Either way, solvable and encryption is important. Badly designed systems and or lack of tooling isn't really an encryption problem.
Anyway, VMs should not have authentication, it makes access sooo much easier. Also drop your IPs while you're at it. Might be useful for debugging later.
Unfortunately with SSH specifically, the dissectors aren't very mature - you only get valid parsing up to the KeX completion messages (NEWKEYS), and after that, even if the encryption is set to `none` via custom patches, the rest of the message flow is not parsed.
Seems because dumping the session keys is not at all a common thing. It's just a matter of effort though - if someone put in the time to improve the SSH story for dissectors, most of the groundwork is there.
Interesting, I thought it was possible to decrypt SSH in Wireshark a la TLS, but it seems I'm mistaken. It still would have been my first goto, likely with encryption patched out as you stated. With well documented protocols, it's generally not too difficult deciphering the raw interior bits as needed with the orientation provided by the dissected pieces. So let me revise my statement: this probably would have been a fairly easy task with protocol analysis guided code review (or simply CR alone).
It all depends on the key exchange mechanism (KEM) used at the start of the TLS session. Some KEM have a property called “perfect forward secrecy” (PFS) which means it’s not possible to decrypt the TLS session after the fact unless one of the nodes logs out the session key(s). Diffie Helman and ECDH are two KEM that provide a PFS guarantee.
Sure it could have been, if you knew about SSH packet inspectors in Wireshark...
The author didn't, and used a general tool to their aid - why is that unfortunate?
Asking an LLM about SSH (hint: the two S-es stand for security) would tell you why only having packet capture in Wireshark isn't going to reveal shit.
Not even remotely accurate. While the dissector is not as mature as I thought and there's no built-in decryption as there is for TLS, that doesn't matter much. Hint: every component of the system is attacker controlled in this scenario.
> Not even remotely accurate.
> there's no built-in decryption
Is that because wireshark can't do that just from packet captures?
1 reply →
Wireshark can decrypt it, so I don't understand what you mean?
Not from packet captures, it can't.
obviously OPs empirical and analytical rigor are top notch. He applied LLMs in the best way possible: fill gaps with clumsy command line flags or protocol implementations. Those aren't things one needs to keep in their head all the time.
Way to gatekeep. God forbid people use tools to help them investigate instead of knowing the exact approach to take.
My thoughts exactly. The OP used AI to get a starting point to their investigation, then used their skills to improve their game, with actual (I guess according to the article itself) proof of that, as opposed to just approving changes from the LLM.
This looks like an actual productivity boost with AI.
What I suggested (mistakenly so, see my revised suggested approach in response to one of your siblings) is the exact opposite of gate keeping.
ChatGPT gaslit the OP telling it there was no such thing as keystroke chafing. So yes, in this case it would have been better to do the work oneself.
How much are you staking on that bet?
Well, I spent a good part of my career reverse engineering network protocols for the purpose of developing exploits against closed source software, so I'm pretty sure I could do this quickly. Not that it matters unless you're going to pay me.
So you are basically overqualified to tell other people how to do it, especially with the payment part.
5 replies →
Sigh.
I'm still waiting for a systems engineering tool that can log every layer, and handle SSL the whole pipe wide.
Im covering everything from strafe and ltrace on the machine, file reads, IO profiling, bandwidth profiling. Like, the whole thing, from beginning to end.
Theres no tool that does that.
Hell, I can't even see good network traces within a single Linux app. The closest you'll find is https://github.com/mozillazg/ptcpdump
But especially with Firefox, good luck.
Real talk though, how much would such a tool be worth to you? Would you pay, say, $3,000/license/year for it? Or, after someone puts in the work to develop it, would you wait for someone else to duct tape something together approximately similar enough using regexps that open source but 10% as good, and then not pay for the good proprietary tool because we're all a bunch of cheap bastards?
We have only ourselves to blame that there aren't better tools (publicly) available. If I hypothetically (really!) had such a tool, it would be an advantage over every other SRE out there that could use it. Trying to sell it directly comes with more headaches than money, selling it to corporations has different headaches, open-sourcing it don't pay the bills, nevermind the burnout (people don't donate for shit). So the way to do it is make a pitch deck, get VC funding so you're able to pay rent until it gets acquired by Oracle/RedHat/IBM (aka the greatest hits for Linux tool acquisition), or try and charge money for it when you run out of VC funding, leading to accusations of "rug pull" and development of alternatives (see also: docker) just to spite you.
In the base case you sell Hashimoto and your bank account has two (three!) commas, but worst case you don't make rent and go homeless when instead you could've gone to a FAANG and made $250k/yr instead of getting paid $50k/yr as the founder and burning VC cash and eating ramen that you have to make yourself.
I agree, that would be an awesome tool! Best case scenario, a company pays for that tool to be developed internally, the company goes under, it gets sold as an asset and whomever buys it forms a compnay and tries to sell it directly and then that company goes under but that whomever finally open sources it because they don't want it to slip into obscurity but if falls into obscurity anyway because it only works on Linux 5.x kernels and can't be ported to the 6.x series that we're on now easily.