Comment by hadlock
4 days ago
Git ops (declarative infra and config) and containers, make automation really, really easy, and completely eliminate all sorts of classes of problems. We push code to prod dozens of times a day without any issues, for months and months at a time. Typically backed by only a single devops engineer to keep everything humming along or building new automation. The automation is the clear process, spitting out messages regularly to email or group chat somewhere. And provides the audit trail.
I worked at a traditional finance company and we had a team of 8 people in traditional operations and another 30 people doing manual testing around the clock to support about 20 developers, 10 network staff, plus another 20-30 managers or leads and security. We could only deploy once a week and there were always issues with "final check out" on sunday morning when hotfixes had to go in or config was modified.
, sorry to laugh but I’ve worked a lot of fintech and this is so painfully on point. I’ve also experienced the nirvana of a mature gitops system - getting there is really painful though (IMO)
the funny thing is, the fintech company in the example you gave likely sees nothing wrong with this. I’ve seen cases where the release cycle is once a month or longer, similar team sizes, and they don’t think they have an issue and would probably laugh at you or look at you weird if you mentioned ci/cd.