Comment by londons_explore
4 hours ago
> access "in case they need it"
A good solution to this is to have a "break glass" system. Ie. access permissions get granted to the developer directly in an emergency by going through a set of steps available beforehand. All requests by that developer are then allowed, limited to 10 kilobytes of results per minute and logged.
For example, imagine you run a bank, and a rare code bug means that at exactly midnight on new years eve all bank balances have $1B added to them.
Your oncall takes down the production servers to to reduce losses and uses the break glass system, by getting approval from n other developers, allowing them direct database access. They use that access to count(accounts) with a balance around $1B or more, see it is everyone, and subtract $1B from them all, before bringing production back up with an older software release. Downtime = 2 hours.
In the morning, all the developers actions are reviewed.
If the developer turned evil and trying to dump the database, the n approvers and 10 kilobytes per minute would reduce data loss. But the break glass system still lets the developer make fairly major emergency fixes to reduce losses or downtime.
Break glass systems are very good but they are also challenging to make. It requires at minimum a system to get approvals from n other people and a system to do logging. It means your authentication and authorization systems must both be working, at least a small amount of non-volatile storage is available, and this often implies the availability of more foundational services such as service discovery, name lookup, distributed consensus, and replicated storage.
The problems can be overcome, but management types may balk at the cost of a complete break glass system.
In your scenario anyone who had $1B or more before midnight is now $1B poorer, so I don't think people would be very happy with that engineer either, regardless of whether there's a break glass system.