Comment by alok-g
19 hours ago
Any guidelines on how to back up such that the attacker cannot reach (when the hacker otherwise had some valid credentials)?
19 hours ago
Any guidelines on how to back up such that the attacker cannot reach (when the hacker otherwise had some valid credentials)?
You can use old school way. You can have a tape backup, ssd, or a device that is not connected to the network constantly. Copy there and unplug the device.
This is how we implemented this at our company:
- We have 2 sources of data that we must backup to continue existing as a business; our postgres and binary files in S3. Everything else is derivable (elasticsearch, so on).
- For postgres, we use barman. With the help of opus/fable, you can get a streaming replication backup working in no time. We have one into another server in the same datacenter (we use baremetal) and another one in another server in a different datacenter.
- We then have a last resort barman backup with bi-weekly base backups + WAL streaming to S3 (both the base backup and WALs). It sends these backups + wal segments into an specific S3 bucket that has object lock in compliance mode. This is a feature from AWS S3 that even the most privileged account credentials (super admin) can't turn off nor delete the files before the object lock, which is 10 days in our case. Object lock compliance mode can only be extended, never shortened.
- For S3, we store them into another versioned bucket, with lifecycle rules to also expire non current versions (== deleted objects) after 10 days. No point in object lock compliance here because it would only protect objects for the most recent 10 days, and you gain nothing. What we do instead: the app servers only have access to these bucket tru an IAM credential that can't delete old versions (so deleted objects have to expire manually via the lifecycle rule) AND this IAM credentials also can't change the object policy.
IMHO, this protects us enough so that even in the worst case scenario (ransomware) we have 10 days to sort everything out and recover our AWS access.
And yes, we test the S3 barman restoration and it works fine. Data loss is at max 5 minutes due to the archive_timeout=300s on the primary.
For the streaming replications in the two servers I mentioned, it's less <1ms, but those wouldn't protect us much in the case of the ransomware - even tough we use tailscale and one compromised server can't ssh into the other.
You need something that's not online or at least not with credentials and devices the Devs and operations people use.
The worst case is the company itself denying you access. More likely to happen with Google.
So if a hacker infiltrated your system and silently fed plausable but wrong data into your databases for 10 days, your whole system would be screwed?