Comment by throwanem
6 years ago
Github is as much or more a collaboration tool as a Git upstream. The Git part is easy to distribute. The collaboration tool, not so much.
And to anticipate the likely next argument - no, mailing lists are not a better tool for collaboration. They are distributed, sure. In so saying, I have exhausted the list of their virtues.
The "we deploy production directly from Github" pathology is the one I'm talking about. Not being able to "collaborate" for a brief outage is fine. Not being able to deploy code isn't.
You gotta CI/CD from somewhere. Whatever that somewhere is, it can go down.
You can, of course, override your CI/CD and do a manual deploy. It's not a matter of "can." It's a matter of not fully understanding all the checks the CI/CD system does to the code, and all the build steps for prod builds, and therefore not having the confidence to deploy to prod without CI/CD holding your hand. (Which I don't at-all blame devs in bigcorps for not knowing; release management is its whole own thing, and division of labor means that at sufficient scale it doesn't make sense to learn about it.)
Well, the place I run my CI tests is on a VMware virtual machine, e.g.
If the tests are automated enough to run with every Git checkout, they can be easily enough be run by hand too.
The real world script I use is a little more complicated, because the code base in question is two decades old so I need to make some changes to how the code looks to the tests so that the tests can run.
I once worked for a company which had a series of tests which took eight to ten hours to run. Running those tests with every single Git checkin was out of the question; we instead used cron to run the tests every night.
Part of the point of having release infrastructure is so that every engineer doesn't have to know how to manage a deployment, even. Having worked both with and without it, I have to say I favor the former. The occasional loss of productivity due to a service provider outage, or even due to whatever the hell's going on today, is vastly outweighed by the day-to-day baseline requiring next to no time spent on the mechanics of taking code from review approval to live in prod.
As a side note, gotta say, scare-quoting "collaborate" seems like a pretty weird flex in a time when everybody's stuck working from home whether they prefer it that way or not. Maybe you prefer to work entirely within a silo, rarely dealing with colleagues in the course of your day-to-day. Most don't, nor should they. Most kinds of work, even the kinds of work that we do, really aren't better done that way.
I worked in a silo for years, not so much by choice as by virtue of the fact that my company at that time could not afford another engineer of similar capability. I got by well enough, thanks in no small part to a youthful talent for improvisation, and the fact that things were simpler then. I wouldn't go back to that life if you paid me. But even if I would, I couldn't. Times change, and the time of the engineer as hermit has largely passed. There are things about it I miss, sure. But I don't miss them so much that I want to imperil my ability to keep earning a living by ignoring that the world has moved on.
edit: Of course, this ignores the difference between modern dev work, and modern sysadmin (as opposed to devops) work. It probably doesn't make much sense to analyze the latter as though it were the former. Or vice versa.
In my experience, there are a lot of developers who get really uncomfortable once they start having more developers working on their code. There’s a lot of effort which has to be done documenting the code: Either by explaining how the code runs to other engineers on the team, or by having documentation showing how the code is arranged and runs, something a lot of engineers are not very good at doing.
It takes a lot of talent to do collaboration well, because it means having to let go of the code being “my code”, and it means having a lot of people skills, such as the ability to empathize with how someone not familiar with the code will see things, and how to write documentation about the code structure so that things can be in maintainable “boxes”.
1 reply →
You read a lot more into that that I intended.
I'm suffering some fatigue w/ the word "collaborate" being thrown around in marketing collateral right now. You used the word several times, and I engaged my "marketing avoidance" mode.
1 reply →
Not sure why you're getting downvoted, but yes, there are plenty of businesses that can't afford an outage in the push-to-prod path.
But it might be worth keeping in mind that there are plenty of businesses that can afford afford delays, especially if they can save money in exchange for that higher risk.
As always, engineering isn't about the perfect solution for everybody, but the right tradeoffs for your particular use case.
All those deployments typically rely on:
1. Scripts available in the git repository.
2. Parameters that are either stored (possibly encrypted) in a filesystem, on a parameter server (like AWS SSM key manager), or in 1Password/LastPass.
It's only a pathology if a team couldn't, as a worst case, coordinate a deployment from a single machine over the phone.