Comment by echoangle
6 hours ago
If a compromised laptop was the entry point and that allowed someone to steal 6 billion dollars, it’s not really the engineers fault, right?
6 hours ago
If a compromised laptop was the entry point and that allowed someone to steal 6 billion dollars, it’s not really the engineers fault, right?
"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars." - Weinberg’s 'First Principle of Financial Management' and 'Second Rule of Failure Prevention’ - 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg
Well it is engineers fault. But not that engineers. Some engineer designed a system without extreme oversight that should be involved in the process.
If you follow that trail far enough it's whoever designed cryptocurrency, right? Because banks can and do reverse transactions as last resort.
No, because this is explicitly the design of Bitcoin. If you give me a foo machine but it makes a bar, that's your fault. If you give me a foo machine and the foo hurts me, that's my fault.
10 replies →
Banks can't reverse wire transfers. Unless there is some super-level intervention that is very rarely used. But they can trace where it went.
This kind of low bar reminds me of FTX
Did they get enough time to design and test in properly?
If nothing else it shows a complete disregard for insider threat.
Nope, that engineer had far too much access. I'm confident this is the case in pretty much all organizations to some degree, but a lot of engineers are also the only ones that can deny themselves the access so in a roundabout way it's their fault anyway.
But a lot (generalizing here) don't like giving up access "in case they need it". I haven't worked backend in ages, but years ago there was one developer that went to the ops guys and said "can I just ssh to production I need to check some env vars to debug my software" (or something to that degree). Ops team was good and it was a hard no - "just add some logging, you can deploy a new release quickly enough".
It's not a access thing, it's a security control.
For example at most good crypto companies that hold wallets it's not possible to ssh at all, even if someone asked and go permission. It just simply isn't possible short of having a AWS root key or physical access to a computer in a DC.
There is no login, no accounts, no ssh keys or similar, they are immutable containers running from images with a signed SHA. The SHA can only be built by a build system that has it's own set of strict security controls.
Without details, I don't know why a comproised laptop would make any difference unless it submitted code through a PR and had it approved by multiple people.
> access "in case they need it"
A good solution to this is to have a "break glass" system. Ie. access permissions get granted to the developer directly in an emergency by going through a set of steps available beforehand. All requests by that developer are then allowed, limited to 10 kilobytes of results per minute and logged.
For example, imagine you run a bank, and a rare code bug means that at exactly midnight on new years eve all bank balances have $1B added to them.
Your oncall takes down the production servers to to reduce losses and uses the break glass system, by getting approval from n other developers, allowing them direct database access. They use that access to count(accounts) with a balance around $1B or more, see it is everyone, and subtract $1B from them all, before bringing production back up with an older software release. Downtime = 2 hours.
In the morning, all the developers actions are reviewed.
If the developer turned evil and trying to dump the database, the n approvers and 10 kilobytes per minute would reduce data loss. But the break glass system still lets the developer make fairly major emergency fixes to reduce losses or downtime.
Break glass systems are very good but they are also challenging to make. It requires at minimum a system to get approvals from n other people and a system to do logging. It means your authentication and authorization systems must both be working, at least a small amount of non-volatile storage is available, and this often implies the availability of more foundational services such as service discovery, name lookup, distributed consensus, and replicated storage.
The problems can be overcome, but management types may balk at the cost of a complete break glass system.
In your scenario anyone who had $1B or more before midnight is now $1B poorer, so I don't think people would be very happy with that engineer either, regardless of whether there's a break glass system.
Not only. I'd say the fact that something was stolen might be the engineers fault (depending on how exactly the laptop was compromised). But the amount that could be stolen like that is just ludicrous.
If you download a typosquatted package because you don't check what you type or your AI assistant got fooled I'd consider it my fault in such a case.
If the engineer, without being hacked, could take 5 billion dollars from your company and flee the country to go buy an extradition-free island, you have already gravely fucked up as an enterprise. The only way this happens is if your system is designed to allow this to happen.
This is how virtually all "crypto" exchanges work -- especially the closer you get to "shitcoin casino". Proper role and system isolation is a challenge if you support just one cryptocurrency. It would be incredibly hard for dozens, assuming service supporting dozens bothered.
I agree it's a grave fuckup but its a common one. You should just assume that when you deposit coins somewhere there are dozens of people who could just walk off with everything on their own. It's remarkable that it doesn't happen more often.