Comment by davedx
2 years ago
All that infra doesn’t integrate itself. Everywhere I’ve worked that had this kind of stack employed at least one if not a team of DevOps people to maintain it all, full time, the year round. Automating a database backup and testing it works takes half a day unless you’re doing something weird
Setting up a multi-az db with automatic failover, incremental backups and PiTR, automated runbooks and monitoring all that doesn't take half a day, not even with RDS.
No, but again, that sounds like a lot of complexity your average startup does not need. Multi-az? Why?
Because their Enterprise client requires it on their due diligence paperwork.
1 reply →
A startup sized company using this many tools? They're for sure doing something weird (and that's not a compliment :) )
Totally on your side with this one - but alas, people associate value with complexity.
> Automating a database backup and testing it works takes half a day unless you’re doing something weird
True story bro
I'm sure that's possible if you're storing the backup on the same server you're restoring on and everything is on top of the line nvme storage. Otherwise your backup just started to run and will need another few days to finish. And that's only if you're running single master.
You're massively underestimating the challenge to get that kind of automation done in a stable manner - and the maintenance required to keep it working over the years.
I’ve implemented such a process for companies multiple times, bro. I know what I’m talking about.
And that's the problem. "It's easy for me because I've done it a dozen times so it's easy for everyone" is a very common fallacy.
5 replies →
I've done it before,too. For toy project, it's easy as you said. It's not once you're at scale. It's hilarious that people are down voting my comment. I guess there are a lot of juniors suffering from the dunning Kruger syndrome around right now
I worked at a place with its own colo where they ran several multi TB MySQL database servers. We did weekly backups and it could take days. Our backups were stored on external USB disks. The I/O performance was abysmal. Taking a filesystem snapshot and copying it to USB could take days. The disks would occasionally lock up and someone would have to power cycle them. Total clown show.
I would rather pay for RDS. Databases are the one thing you don't want to screw up.