Comment by madradavid
5 hours ago
Looks like a DevOps guy is going to take the blame for this: "Binance's initial investigation suggests that the security incident may involve a compromised DevOps engineer’s laptop, which allowed an attacker to gain access to a service container and submit unauthorized withdrawal transactions." https://x.com/binance/status/2115993361517891592
If a compromised laptop was the entry point and that allowed someone to steal 6 billion dollars, it’s not really the engineers fault, right?
"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars." - Weinberg’s 'First Principle of Financial Management' and 'Second Rule of Failure Prevention’ - 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg
Well it is engineers fault. But not that engineers. Some engineer designed a system without extreme oversight that should be involved in the process.
If you follow that trail far enough it's whoever designed cryptocurrency, right? Because banks can and do reverse transactions as last resort.
10 replies →
This kind of low bar reminds me of FTX
If nothing else it shows a complete disregard for insider threat.
Nope, that engineer had far too much access. I'm confident this is the case in pretty much all organizations to some degree, but a lot of engineers are also the only ones that can deny themselves the access so in a roundabout way it's their fault anyway.
But a lot (generalizing here) don't like giving up access "in case they need it". I haven't worked backend in ages, but years ago there was one developer that went to the ops guys and said "can I just ssh to production I need to check some env vars to debug my software" (or something to that degree). Ops team was good and it was a hard no - "just add some logging, you can deploy a new release quickly enough".
> access "in case they need it"
A good solution to this is to have a "break glass" system. Ie. access permissions get granted to the developer directly in an emergency by going through a set of steps available beforehand. All requests by that developer are then allowed, limited to 10 kilobytes of results per minute and logged.
For example, imagine you run a bank, and a rare code bug means that at exactly midnight on new years eve all bank balances have $1B added to them.
Your oncall takes down the production servers to to reduce losses and uses the break glass system, by getting approval from n other developers, allowing them direct database access. They use that access to count(accounts) with a balance around $1B or more, see it is everyone, and subtract $1B from them all, before bringing production back up with an older software release. Downtime = 2 hours.
In the morning, all the developers actions are reviewed.
If the developer turned evil and trying to dump the database, the n approvers and 10 kilobytes per minute would reduce data loss. But the break glass system still lets the developer make fairly major emergency fixes to reduce losses or downtime.
It's not a access thing, it's a security control.
For example at most good crypto companies that hold wallets it's not possible to ssh at all, even if someone asked and go permission. It just simply isn't possible short of having a AWS root key or physical access to a computer in a DC.
There is no login, no accounts, no ssh keys or similar, they are immutable containers running from images with a signed SHA. The SHA can only be built by a build system that has it's own set of strict security controls.
Without details, I don't know why a comproised laptop would make any difference unless it submitted code through a PR and had it approved by multiple people.
Not only. I'd say the fact that something was stolen might be the engineers fault (depending on how exactly the laptop was compromised). But the amount that could be stolen like that is just ludicrous.
If you download a typosquatted package because you don't check what you type or your AI assistant got fooled I'd consider it my fault in such a case.
If the engineer, without being hacked, could take 5 billion dollars from your company and flee the country to go buy an extradition-free island, you have already gravely fucked up as an enterprise. The only way this happens is if your system is designed to allow this to happen.
1 reply →
I recommend everyone to move your plantext secrets from ~/.kube/config, ~/.ssh etc into a password-manager. You can tell Claude to adjust the config files to get the secrets from 1Password when they are needed with TouchID requirements etc.
These days I’m getting so many 1Password prompts for agents pulling env variables out of it and with very little context of what is being accessed by who (The dialog 1Password shows is pretty bad) that it would be very easy for something to slip in there.
> I’m getting so many 1Password prompts for agents pulling env variables out of it and with very little context of what is being accessed by who
This is a one-sentence horror story. I almost can't believe someone would let this happen, unless the 1Password account is not important (highly unlikely).
2 replies →
I have been using Yubikeys for years, according to this guide: https://github.com/drduh/yubikey-guide
The nice thing is that using the private key requires physical touch. That means that AI itself (or a remote attacker, for that matter) can't do things unless I confirm.
if $5.6B is protected by your index finger then you're gonna need kidnap insurance, and a really good pair of gloves.
2 replies →
How would that help when your workstation is compromised? They can just steal your authentication the moment you log in through the password manager
It does help in that the secrets are not in plain-text, readable for any script you run. With 1Password, a process asks for the password and it gets transferred not through the filesystem but through IPC. 1Password shows a dialog: "Do you want iTerm to get your SSH key ABC123?"
In my thinking, this is more secure than plaintext in filesystem
2 replies →
My secret is `4`. Is that ok? Do I still need to move it??
4 is fine, but only if you’re a certificate authority.
as for ssh keys, you can generate your keypair in secure enclave and reuiqre touchId for every new ssh session. The private key never leaves the secure enclave.
Probably the better option instead of relying on a password manager.
1 reply →
Isn’t the password manager a prime target too?
you know most password managers have open sockets, or other channels to other local processes to interact with and allow stealing entries?
you know most tools like gcloud, bitwarden, etc store auth keys just in dotfiles too?
Local compromise is KO now, and we can't do shit on platforms like MacOs didn't research Linux recently
The fact a single devops guy has the access to do this is the actual issue
They claim the attack wasn't just "compromised laptop ==> drain wallets" but had more steps in-between, with the attackers using image registries as an entry point into their systems.
One person's key probably should not be enough to move billions no matter who it is.