Comment by Cthulhu_
3 hours ago
Nope, that engineer had far too much access. I'm confident this is the case in pretty much all organizations to some degree, but a lot of engineers are also the only ones that can deny themselves the access so in a roundabout way it's their fault anyway.
But a lot (generalizing here) don't like giving up access "in case they need it". I haven't worked backend in ages, but years ago there was one developer that went to the ops guys and said "can I just ssh to production I need to check some env vars to debug my software" (or something to that degree). Ops team was good and it was a hard no - "just add some logging, you can deploy a new release quickly enough".
> access "in case they need it"
A good solution to this is to have a "break glass" system. Ie. access permissions get granted to the developer directly in an emergency by going through a set of steps available beforehand. All requests by that developer are then allowed, limited to 10 kilobytes of results per minute and logged.
For example, imagine you run a bank, and a rare code bug means that at exactly midnight on new years eve all bank balances have $1B added to them.
Your oncall takes down the production servers to to reduce losses and uses the break glass system, by getting approval from n other developers, allowing them direct database access. They use that access to count(accounts) with a balance around $1B or more, see it is everyone, and subtract $1B from them all, before bringing production back up with an older software release. Downtime = 2 hours.
In the morning, all the developers actions are reviewed.
If the developer turned evil and trying to dump the database, the n approvers and 10 kilobytes per minute would reduce data loss. But the break glass system still lets the developer make fairly major emergency fixes to reduce losses or downtime.