Comment by bouk
5 hours ago
Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...
It's funny, when we were acquired they started moving us to Github actions but it seems that maybe we should stay on our old crusty self-hosted Jenkins setup...
You should never entirely depend on a third party service for deployments.
Been burned too many times on that one.
Ok.
Move to EC2.
Darn AWS is down.
Alright, run it on a Mac Mini in your basement. Ahh dawn, your ISP is having issues. Good thing you have a backup 5G hotspot.
Ohh no, the power is out.
Eventually you have to trust someone else.
GitHub is a tragedy of the Commons. Too many people are using it, and Microsoft isn't willing to handle it correctly.
Feels like a very good business opportunity. Minimum 50k yearly contracts, GitHub with actual uptime. GitPro ?
We’re actually moving back to redundant data centres due to all of those problems.
Aggregate risk is too high.
It's almost as though GitHub should never have let itself be sold to Microsoft...
1 reply →
Maybe we need a split between source management and distribution? The former looks like git[hub] to me, the latter maybe more like a Linux distro repo?
We could still deploy manually but it's suboptimal! And we're 'flying blind' without CI runs
> And we're 'flying blind' without CI runs
You should never entirely depend on a third party service to run your tests, either.
Same thoughts - we use an action to ship to production, its builds an image, pushes it to ECS which triggers a deployment.
We can't be blocked here. Seems silly what we settled on this, but for a long time GitHub had been reliable enough for many years, but things are sliding down the pan as of late.
Sounds like a very easy process to rewrite in bash/python and have it on hand if needed.
./deploy.sh
It is a control pain
It's always best to be portable - always be able to do builds and releases locally (at least, once you get the keys - it shouldn't be possible by default), then add things like github actions on top as convenience.
Self host gitlab. If you already host runners it’s not a big lift.
Same here. You’d think they could at least separate out the GitHub-hosted and self-hosted runners, so you’re still able to dispatch jobs if the self-hosted runners are down.
If the job queue is down, that wouldn't help, would it?
On my repo the jobs do not get scheduled on the PRs at all, so I assume that separation wouldn't help for todays issue.
They have the github enterprise domain separated out and its working fine right now https://us.githubstatus.com/posts/dashboard
1 reply →
Depending on how many thousands of $ per year, it would probably be cheaper and more reliable to self-host GitLab. It's better in terms of organisational structure (you can have one, including access and secret inheritance), and (personal view) Gitlab-CI is better than GitHub Actions because it doesn't push you towards a JavaScript/NPM style dependency hell. And it's actually fairly easy to self-hosted, with options from a single machine with an omnibus package that handles everything to a full blown autoscaling Kubernetes deployment.
Sounds good until you see their cvedetails page
Hide it behind VPN, so it's not accessible from outside.
When you own it you can just limit it into vpn-ed company users, that significantly cuts down on the area that can be hit
I mean, the GitHub Actions supply chain risks and attacks definitely compensate for any GitLab security vulnerabilities you can think of.
1 reply →
> For a service we're paying thousands of $ per year for even though we host runners ourselves...
Wait until you charge you for self-hosting runners.
Oh wait. They already tried.
Sure. Don't use GitHub.
You can now hire me as an overpriced consultant instead of paying Microsoft.