Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure?
It might also be covid related. People are working from home, people responsible for system upkeep might not be immediately responsive, more demand on the servers for whatever reason, etc.
I would agree that the coronavirus could be a factor here. At the same time, I've been noticing issues since probably December or January (before the coronavirus started being a real problem), which makes it seem like maybe there are multiple issues.
Of course, I'm not actually internal to Microsoft or Github, so I have no idea and it's all opaque to me.
Maybe it's the demand side? With remote work, the intensity of usage went up at least in our company as we rely more on written communication. At the same time, some people will use the time to start side projects or get into programming.
Personnel may be partially unavailable, but - home or office - developers are doing the same job as they always did. At least from the users point of view it didn't change that much.
Ideally GitHub Actions would be completely independent of GitHub's core services/servers (e.g. how Travis CI, Circle CI, etc. works), but that seems like it may not be the case.
Also, I'm still anticipating a fuller report on the database issues they mentioned have been the root cause of many outages over the past few months.
I prefer to pay 8$ per month for a stable service any time over 4$ per month over a service that fails during a critical build, just like it happened to me this time.
If you view the historical uptime, it does seem like there have been more incidents in the past three months, but otherwise the waters look calm (as reported at least): https://www.githubstatus.com/uptime?page=1
Yes, and so this is actually a thing that bugs me, because I use Github every day. Over the past several months, there have been numerous days in which I've had problems (`You can not comment at this time`, 500s, etc.) and no corresponding status report.
It seems like the historical uptime page paints a far rosier picture than I am actually experiencing.
If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here:
It's unfair to Github to make the claim that your infra is more resilient and better maintained. Their load is orders of magnitude greater than yours. My driveway also doesn't have potholes it doesn't mean it's more resilient than the freeway.
Sidebar, I just want to say, you are one of the few people I’ve observed doing actual “modern” web development.
When most people talk about “modern web” or modern anything in software they think it means “using all the latest tools”.
That often means things like ES6 and Webpack, which have nice surfaces, but which create nightmares under the hood.
That’s the opposite of what modern architecture was. It was about embracing the constraints of materials. Given the properties of concrete, what is the limit of what you can do with it. Go there, and no further. And don’t cover it up, just finish the dang slab and get on with the rest of the house.
ES6 means transpiling, which means webpack, which means a massive machine of hidden complexity, which if you’re lucky exposes a nice smooth surface where everything is arrow functions and named exports. And if you’re unlucky is a flimsy piece of cardboard over the nightmare underneath.
You (SourceHut) seem to be building a UI that actually takes note of how the browser is. And you are trying to push the big numbers... how reliable your service can be, how many endpoints can one person maintain, while letting the materials of the web (forms, urls) dictate the details.
That’s true modernism.
So, bravo. I’m glad to see you out in the world. It takes courage to step outside of the norm and I’m rooting for you.
Thanks for being so transparent about your operations.
Maybe this is somewhere in the manual and I missed it, but do you have some way of automating the configuration of your hosts and VMs? For example, do you use something like Ansible?
I set up a self-hosted Gitea this year and moved my repos over and couldn't be happier with it. It's faster than GitHub, clones the GitHub design/UI so that everything's where I expect it to be, has a dark mode, and supports U2F. It's easy to deploy, back up, and maintain, the Gitea devs have done a great job.
It's much less complicated (both from an admin standpoint, as well as a UI standpoint) than GitLab. I paired it with a Drone installation (also self-hosted) for CI and (sometimes) CD.
It all works great, and is way easier than I thought. If there's downtime, I'm (usually) in control of when or how long, as I have root on the box.
I'm also not giving my money to a giant military contractor (Microsoft, the owners of GitHub) any longer, which is a huge deal for me from a personal moral standpoint (YMMV).
Positive side for such downtime, turns out people gather to look at your landing/home page now just to see if the service is up. Can the cost of a downtime be effective to grab few customers for your new feature just published on your homepage?
you could move to gitlab, but from what im hearing the pricing is higher than github (is this still true?)
Barring that you always have the tried and true (and for some reason abhorred by start-ups) option of running your own gitea or gitlab instance. Its not hard, and most of this stuff can be done in dockerless containers if you want.
If cloud servers are getting "overloaded" as some commenters say, you could even buy a few racks or u's of colo somewhere or use a cloud provider that isnt the most popular meme on YC. Vultr and ramnode are both good options and youd be supporting a small business, not Bezos next giga-yacht.
>Vultr and ramnode are both good options and youd be supporting a small business, not Bezos next giga-yacht.
vultr is considered a small business now? crunchbase lists them as having 50-100 employees, and they seem to be owned by Choopa, LLC which some sources list as having 150 employees.
Congrats, you mentioned everything that's less likely, even came up with a conspiracy theory but managed to avoid mentioning the most logical explanation (increase in the use of existing repos, new user growth and bandwidth scarcity that can't be fixed within a short timeframe).
1) Microsoft took over
2) M$ migrates some ADO (Azure DevOps) features to Github (e.g., Github Actions)
3) If Github was not on Azure before M$ bought it (very likely, but needs citation) they will probably migrate to Azure at some point
I'm pretty sure Github Actions work predates the MS acquisition... I'm also pretty sure that they are trying to align the backend systems more to Azure, but have no insight into how much of that took place.
The fact that you used "M$" indicates that you are predisposed to blame Microsoft for actions that are likely not from the parent, and discount any changes from the top down that have occurred within MS. And while I have a lot of issues with MS and Windows in particular, MS today is not the same as MS even a decade ago.
I would guess that since they introduced free private repos, usage has been increased a lot.
Eg. I used to use bitbucket but switched over to github when they did that, cus the github desktop program is nice and works a lot more smoothly with gtihub as opposed to bitbucket.
I don't think it's just GitHub. I noticed everything seemed slow but thought it was just my ISP. Turns out from looking at downdetector.com there seems to be a large spike in reported problems across many sites and providers, all occurring at the same time.
Yeah apparently Centurylink and Comcast are having issues as well. Customers have been complaining about slow load times with our products today. Coworkers complaining about slow load times in general and especially with Salesforce. I’m being rerouted from my local CloudFlare data center to Chicago as well.
Not entirely sure what’s up, but this doesn’t thing like an issue limited to only GitHub.
I thought this at first too but downdetector sorts all the problem sites to the top which makes it look like the internet is melting when you first look at it. It actually seems reasonable for this many sites to be having issues at any one given time.
Here in Minnesota, several coworkers and I are having trouble with our ISP. And a website we host is (apparently) being DDoSed. And now GitHub's down, and you're reporting some Azure issue. Is something going on...?
No, I don't think so (and I'm also in Minnesota). I'm going to guess that the increased load is just pushing services over the edge.
Edit: Also, interestingly enough, I am now reliably hitting Cloudflare's ORD (Chicago) datacenter instead of MSP. If you visit https://snazz.xyz/cdn-cgi/trace (or any other Cloudflare-backed website), what comes after the COLO= for you?
Or their data centers are just full. Teams had a tremendous growth, so did their O365 suite and probably also Azure itself. At some point even the largest vendors run out of servers.
In my experience, nothing is correlated more with downtime than code changes. Github has been pushing a lot more features since the Microsoft acquisition, and has felt down a lot more often since then.
There has been one major outage a month the last several months, with sprinklings of little outages (I recall webhooks down quite a lot). The cadence of these outages is rapidly changing my perception of Github as a reliable service.
At my organization, we saw an uptick in timeouts spanning vendors that started at the same time - 6:55am PST. Makes me think there's an internet wide event occurring right now
Today marks the 3rd time I've broached the topic w/ management of getting the self-host enterprise option... Compared to (public) GitHub's problems this year so far, our AWS EC2 instances are orders of magnitude more reliable.
Sure, the internet can still go down, but my VPN into us-east-1 from Texas has been unbroken for weeks now.
At this point I'd almost prefer to pull it all in-house and manage it myself so the entire team doesn't have to lose a whole day of productivity over all this. I am so glad we moved away from using GH Actions for builds because we would be absolutely hosed right now on supporting our customers.
I'm not sure the irony is lost on you that; people generally prefer services like github (and; similarly EC2) precisely _because_ it's not on-prem and that if it's down there are hundreds of talented engineers working to resolve those issues.
Unfortunately anecdotal experience here is spotty. Services I run for my team are several 9's higher in terms of actual availability (note: I did not say "uptime"); and contrarily other internal services at my company have many times less reliability than services like github.
Given that people prefer external hosting for the reasons I mentioned, for many I think pulling it in-house is unappealing.
Does the GH Enterprise option not provide some degree of support with initial setup and configuration? Is the enterprise support more or less responsive than the public channels? I would expect those same engineers are also responsible for maintaining the enterprise offering.
Also, does isolation of a private GH instance from the public instances not provide some degree of added reliability considering the potential for DDOS or simply extreme load?
I absolutely grant you the IT infrastructure concerns. BUT Amazon is our vendor on that. GitHub provides the software. It's not like I'd be standing up a new series of physical hosts to run an on-prem GitHub built from source and managing all of the hell around that. This would be simply putting a GH-provided image on a EC2 instance and making sure we have frequent snapshots.
I work for a fortune 100 company and GHE has been a nightmare. Not only do you need a team to maintain it, but then that team needs to be equipped enough to help support any other internal services (CI/CD pipelines) that integrate with it. Github.com is just an infinitely more enjoyable experience.
same with Five9 from the KC Metro, also accessing azure resources. No issues on my Google Fiber. But we have agents all over the midwest, all having issues.
It is a tech demo of https://tardigrade.io, because easily storing all of Github certainly gets peoples attention.
Its only expensive because cloud providers overcharge. The listed prices on Tardigrade start at half of Amazon S3, and there are significant discounts if you are storing more than 100 TB.
Why not make two versions of GitHub, one free and one paid, having professional-level uptime and support? Is that the idea behind GitHub Enterprise (https://github.com/enterprise)?
Git pushes are going really slow for me, but they do work after a few tries. If you normally use a credential cache and it's asking you for your password again, hit control-C and retry the operation. That seems to have gotten my last few pushes to work.
What usually happens with CircleCI after a GitHub outage is that they're hit with a flood of GitHub webhooks when GitHub comes back online. Then Circle starts to slow down under the load of all of the new jobs that have been queued.
One usually causes the other, so we shouldn't infer anything larger from that datapoint.
In this case, CircleCI and Github were down at the same time, and seemed to come out of it around the same time too. I don't know what to infer from that either.
Reddit and Rocket League servers were degraded simultaneously yesterday, and if I recall correctly they both host on Google Cloud (I may be mistaken there). My paranoia of a high impact state-sponsored cyber attack have been high!
We use TeamCity and can't say I have any complaints. We utilize the tagging feature pretty heavily in our deployments and source all artifacts from TeamCity builds. The builds also post success or failure to Gerrit.
Github is as much or more a collaboration tool as a Git upstream. The Git part is easy to distribute. The collaboration tool, not so much.
And to anticipate the likely next argument - no, mailing lists are not a better tool for collaboration. They are distributed, sure. In so saying, I have exhausted the list of their virtues.
The "we deploy production directly from Github" pathology is the one I'm talking about. Not being able to "collaborate" for a brief outage is fine. Not being able to deploy code isn't.
Our nonprofit offers both privacy friendly cloud storage and gits with gitlab. We also offer gitlab hosting for private instances. https://git.stealthdrop.cloud and https://my.stealthdrop.cloud. Our service are free and we are 501c3 donations are deductible.
Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure?
...is it time to move away from Github?
It might also be covid related. People are working from home, people responsible for system upkeep might not be immediately responsive, more demand on the servers for whatever reason, etc.
I would agree that the coronavirus could be a factor here. At the same time, I've been noticing issues since probably December or January (before the coronavirus started being a real problem), which makes it seem like maybe there are multiple issues.
Of course, I'm not actually internal to Microsoft or Github, so I have no idea and it's all opaque to me.
2 replies →
Maybe it's the demand side? With remote work, the intensity of usage went up at least in our company as we rely more on written communication. At the same time, some people will use the time to start side projects or get into programming.
2 replies →
Personnel may be partially unavailable, but - home or office - developers are doing the same job as they always did. At least from the users point of view it didn't change that much.
Cloud servers everywhere are also under much heavier load. They may have moved much of it into Azure, which is famously overloaded right now.
The delay/service unavailablity should follow the work at home schedule then- pushes pre-lunch and pre evening dinner.
My guess is opening their paying offering for free and also Github actions which is CPU-intensive and does much more than the traditional CI/CD tools.
Ideally GitHub Actions would be completely independent of GitHub's core services/servers (e.g. how Travis CI, Circle CI, etc. works), but that seems like it may not be the case.
Also, I'm still anticipating a fuller report on the database issues they mentioned have been the root cause of many outages over the past few months.
I prefer to pay 8$ per month for a stable service any time over 4$ per month over a service that fails during a critical build, just like it happened to me this time.
Not just you.
This page https://web.archive.org/web/20190801000000*/https://github.c... has been[1] “500 internal server error” since late-December (globally it seems). Nobody cares (“not a priority” (c) support), nothing on githubstatus.
[1] blue circles on web.archive are errors too
If you view the historical uptime, it does seem like there have been more incidents in the past three months, but otherwise the waters look calm (as reported at least): https://www.githubstatus.com/uptime?page=1
Yes, and so this is actually a thing that bugs me, because I use Github every day. Over the past several months, there have been numerous days in which I've had problems (`You can not comment at this time`, 500s, etc.) and no corresponding status report.
It seems like the historical uptime page paints a far rosier picture than I am actually experiencing.
8 replies →
I wonder if it has something to do with the past 3 months have been coronavirus related, higher internet usage?
They've been pushing lots of features in the platform so not surprising that it's a bit unstable now.
In an ideal world, pushing new features would have no impact on stable mature features like browsing files, comment threads, etc
6 replies →
If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here:
https://man.sr.ht/ops/
It's also the highest performance software forge by objective measures:
https://forgeperf.org/
Full disclosure: I am the founder of SourceHut.
It's unfair to Github to make the claim that your infra is more resilient and better maintained. Their load is orders of magnitude greater than yours. My driveway also doesn't have potholes it doesn't mean it's more resilient than the freeway.
1 reply →
Sidebar, I just want to say, you are one of the few people I’ve observed doing actual “modern” web development.
When most people talk about “modern web” or modern anything in software they think it means “using all the latest tools”.
That often means things like ES6 and Webpack, which have nice surfaces, but which create nightmares under the hood.
That’s the opposite of what modern architecture was. It was about embracing the constraints of materials. Given the properties of concrete, what is the limit of what you can do with it. Go there, and no further. And don’t cover it up, just finish the dang slab and get on with the rest of the house.
ES6 means transpiling, which means webpack, which means a massive machine of hidden complexity, which if you’re lucky exposes a nice smooth surface where everything is arrow functions and named exports. And if you’re unlucky is a flimsy piece of cardboard over the nightmare underneath.
You (SourceHut) seem to be building a UI that actually takes note of how the browser is. And you are trying to push the big numbers... how reliable your service can be, how many endpoints can one person maintain, while letting the materials of the web (forms, urls) dictate the details.
That’s true modernism.
So, bravo. I’m glad to see you out in the world. It takes courage to step outside of the norm and I’m rooting for you.
3 replies →
I'm sure you've done a great job building up your infrastructure, but if you have the level of traffic Github has, what would your uptime be?
16 replies →
Thanks for being so transparent about your operations.
Maybe this is somewhere in the manual and I missed it, but do you have some way of automating the configuration of your hosts and VMs? For example, do you use something like Ansible?
3 replies →
A little bit off topic. I just wondering it support git write access over https? Not just read only.
1 reply →
Yeah I'm sure there are no possible scaling issues between your service (how many people use it / repos acitve) vs github or gitlab....
your $2/month pricing. Is it $2/person/month?
1 reply →
I set up a self-hosted Gitea this year and moved my repos over and couldn't be happier with it. It's faster than GitHub, clones the GitHub design/UI so that everything's where I expect it to be, has a dark mode, and supports U2F. It's easy to deploy, back up, and maintain, the Gitea devs have done a great job.
It's much less complicated (both from an admin standpoint, as well as a UI standpoint) than GitLab. I paired it with a Drone installation (also self-hosted) for CI and (sometimes) CD.
It all works great, and is way easier than I thought. If there's downtime, I'm (usually) in control of when or how long, as I have root on the box.
I'm also not giving my money to a giant military contractor (Microsoft, the owners of GitHub) any longer, which is a huge deal for me from a personal moral standpoint (YMMV).
Positive side for such downtime, turns out people gather to look at your landing/home page now just to see if the service is up. Can the cost of a downtime be effective to grab few customers for your new feature just published on your homepage?
you could move to gitlab, but from what im hearing the pricing is higher than github (is this still true?)
Barring that you always have the tried and true (and for some reason abhorred by start-ups) option of running your own gitea or gitlab instance. Its not hard, and most of this stuff can be done in dockerless containers if you want.
If cloud servers are getting "overloaded" as some commenters say, you could even buy a few racks or u's of colo somewhere or use a cloud provider that isnt the most popular meme on YC. Vultr and ramnode are both good options and youd be supporting a small business, not Bezos next giga-yacht.
Github actually recently reduced their prices to match some of Gitlab's offerings.
https://news.ycombinator.com/item?id=22867627
>Vultr and ramnode are both good options and youd be supporting a small business, not Bezos next giga-yacht.
vultr is considered a small business now? crunchbase lists them as having 50-100 employees, and they seem to be owned by Choopa, LLC which some sources list as having 150 employees.
GitLab community advocate here, just wanted to share the most up to date GitLab pricing information: https://about.gitlab.com/pricing/ Thanks!
It has been since the Microsoft acquisition.
I chalk most of the early ones to moving services over to Azure.
Lately though, I don't know. Azure is running pretty close to capacity, so maybe it's part of the problem
May it has more to do with their changes in pricing and a surge in uptake of customers?
That would point to Azure hosting. Anyone notice a similiar pattern?
2 replies →
As many readers are stating there seem to be larger, internet-wide issues today in US.
For hosting your own repos AWS CodeCommit works very well.
Congrats, you mentioned everything that's less likely, even came up with a conspiracy theory but managed to avoid mentioning the most logical explanation (increase in the use of existing repos, new user growth and bandwidth scarcity that can't be fixed within a short timeframe).
1) Microsoft took over 2) M$ migrates some ADO (Azure DevOps) features to Github (e.g., Github Actions) 3) If Github was not on Azure before M$ bought it (very likely, but needs citation) they will probably migrate to Azure at some point
I'm pretty sure Github Actions work predates the MS acquisition... I'm also pretty sure that they are trying to align the backend systems more to Azure, but have no insight into how much of that took place.
The fact that you used "M$" indicates that you are predisposed to blame Microsoft for actions that are likely not from the parent, and discount any changes from the top down that have occurred within MS. And while I have a lot of issues with MS and Windows in particular, MS today is not the same as MS even a decade ago.
I would guess that since they introduced free private repos, usage has been increased a lot. Eg. I used to use bitbucket but switched over to github when they did that, cus the github desktop program is nice and works a lot more smoothly with gtihub as opposed to bitbucket.
I don't think it's just GitHub. I noticed everything seemed slow but thought it was just my ISP. Turns out from looking at downdetector.com there seems to be a large spike in reported problems across many sites and providers, all occurring at the same time.
Yeah apparently Centurylink and Comcast are having issues as well. Customers have been complaining about slow load times with our products today. Coworkers complaining about slow load times in general and especially with Salesforce. I’m being rerouted from my local CloudFlare data center to Chicago as well.
Not entirely sure what’s up, but this doesn’t thing like an issue limited to only GitHub.
I thought this at first too but downdetector sorts all the problem sites to the top which makes it look like the internet is melting when you first look at it. It actually seems reasonable for this many sites to be having issues at any one given time.
Maybe the cloud providers are running out of capacity?
Maybe The Cloud is full.
1 reply →
Yes, saw similar notice from bitbucket
GitHub is having major availability issues these past three months. 4 incidents in Feb, 4 in March and 6 in April so far.
Source: https://www.githubstatus.com/history
I look forward to reading root cause analysis promised by Nat in Feb: https://twitter.com/natfriedman/status/1233079491204804608
Azure is also refusing to allocate me capacity - I'm wondering if this is a general MSFT outage?
Here in Minnesota, several coworkers and I are having trouble with our ISP. And a website we host is (apparently) being DDoSed. And now GitHub's down, and you're reporting some Azure issue. Is something going on...?
No, I don't think so (and I'm also in Minnesota). I'm going to guess that the increased load is just pushing services over the edge.
Edit: Also, interestingly enough, I am now reliably hitting Cloudflare's ORD (Chicago) datacenter instead of MSP. If you visit https://snazz.xyz/cdn-cgi/trace (or any other Cloudflare-backed website), what comes after the COLO= for you?
6 replies →
Also having erratic few minutes outage from my ISP in the UK, which never happened for years.
Global network issue somewhere?
Also having issues with CenturyLink fiber at home in MN. Stuff is very slow today.
They posted a warning that the uptick in Teams usage and general cloud use has meant they have much less spare capacity - see https://azure.microsoft.com/en-us/blog/update-2-on-microsoft...
Sure, just anecdotally this is my first time running into this so it might be that capacity issues are particularly acute right now.
Even that article states that there hadn't been any service disruptions in the US up to now, but I am certainly experiencing it now.
Or their data centers are just full. Teams had a tremendous growth, so did their O365 suite and probably also Azure itself. At some point even the largest vendors run out of servers.
I had never heard of Microsoft Teams until yesterday when I saw it on a TV commercial.
2 replies →
In my experience, nothing is correlated more with downtime than code changes. Github has been pushing a lot more features since the Microsoft acquisition, and has felt down a lot more often since then.
There has been one major outage a month the last several months, with sprinklings of little outages (I recall webhooks down quite a lot). The cadence of these outages is rapidly changing my perception of Github as a reliable service.
At my organization, we saw an uptick in timeouts spanning vendors that started at the same time - 6:55am PST. Makes me think there's an internet wide event occurring right now
Quite a few issues being reported here: https://www.thousandeyes.com/outages
Which providers are you seeing issues with? I'm curious if we can corroborate.
I was referring to WorldPay and Adyen (payment providers), but also saw issues directly w Github.
We were experiencing issues with Netlify at 6:15am PST.
CenturyLink ISP is having issues: https://downdetector.com/status/centurylink/
Today marks the 3rd time I've broached the topic w/ management of getting the self-host enterprise option... Compared to (public) GitHub's problems this year so far, our AWS EC2 instances are orders of magnitude more reliable. Sure, the internet can still go down, but my VPN into us-east-1 from Texas has been unbroken for weeks now.
At this point I'd almost prefer to pull it all in-house and manage it myself so the entire team doesn't have to lose a whole day of productivity over all this. I am so glad we moved away from using GH Actions for builds because we would be absolutely hosed right now on supporting our customers.
I'm not sure the irony is lost on you that; people generally prefer services like github (and; similarly EC2) precisely _because_ it's not on-prem and that if it's down there are hundreds of talented engineers working to resolve those issues.
Unfortunately anecdotal experience here is spotty. Services I run for my team are several 9's higher in terms of actual availability (note: I did not say "uptime"); and contrarily other internal services at my company have many times less reliability than services like github.
Given that people prefer external hosting for the reasons I mentioned, for many I think pulling it in-house is unappealing.
Does the GH Enterprise option not provide some degree of support with initial setup and configuration? Is the enterprise support more or less responsive than the public channels? I would expect those same engineers are also responsible for maintaining the enterprise offering.
Also, does isolation of a private GH instance from the public instances not provide some degree of added reliability considering the potential for DDOS or simply extreme load?
I absolutely grant you the IT infrastructure concerns. BUT Amazon is our vendor on that. GitHub provides the software. It's not like I'd be standing up a new series of physical hosts to run an on-prem GitHub built from source and managing all of the hell around that. This would be simply putting a GH-provided image on a EC2 instance and making sure we have frequent snapshots.
1 reply →
I work for a fortune 100 company and GHE has been a nightmare. Not only do you need a team to maintain it, but then that team needs to be equipped enough to help support any other internal services (CI/CD pipelines) that integrate with it. Github.com is just an infinitely more enjoyable experience.
There are larger problems I think... our call center is having trouble with Five9 in San Francisco.
I also I tried setting up an EC2 instance in North Cali and just pinging it from Kansas City via Google fiber experienced 37% packet loss.
same with Five9 from the KC Metro, also accessing azure resources. No issues on my Google Fiber. But we have agents all over the midwest, all having issues.
Created https://gitbackup.org for this very reason.
Where do you store all that data?
Should only be 2-3 PB. Storing it all on https://tardigrade.io
6 replies →
Out of curiosity how do you or how to you plan to monetize this? Seems like it would get expensive really quickly.
It is a tech demo of https://tardigrade.io, because easily storing all of Github certainly gets peoples attention.
Its only expensive because cloud providers overcharge. The listed prices on Tardigrade start at half of Amazon S3, and there are significant discounts if you are storing more than 100 TB.
Why not make two versions of GitHub, one free and one paid, having professional-level uptime and support? Is that the idea behind GitHub Enterprise (https://github.com/enterprise)?
Yes. Enterprise has an SLA (when GitHub hosts it) and an option to self-host.
Oh okay, I didn't know GitHub had an option to host Enterprise. (It's way overbudget for me right now, but just curious.)
Who would have guessed that handing control of a company to Microsoft would make it less reliable.
Git pushes are going really slow for me, but they do work after a few tries. If you normally use a credential cache and it's asking you for your password again, hit control-C and retry the operation. That seems to have gotten my last few pushes to work.
got my push to go through, unfortunately downloading packages from https://codeload.github.com was timing out
CircleCI's having a lot of trouble, too. Intermittent 500s, and even the images on the 500 page don't load.
Seems like something bigger might be going on.
What usually happens with CircleCI after a GitHub outage is that they're hit with a flood of GitHub webhooks when GitHub comes back online. Then Circle starts to slow down under the load of all of the new jobs that have been queued.
One usually causes the other, so we shouldn't infer anything larger from that datapoint.
In this case, CircleCI and Github were down at the same time, and seemed to come out of it around the same time too. I don't know what to infer from that either.
I noticed it being more of a centurylink issue.
I think my biggest nightmare is SO, GH, and HN all being down at the same time.
Edit: A nightmare is an exaggeration. But it would slow down work.
Reddit and Rocket League servers were degraded simultaneously yesterday, and if I recall correctly they both host on Google Cloud (I may be mistaken there). My paranoia of a high impact state-sponsored cyber attack have been high!
Yup. Google Cloud seems to be having frequent problems lately.
However, I feel like GitHub being down AND Stack Overflow would slow down work. And HN is a good place to know what's going on.
1 reply →
I think AWS was having issues yesterday and today as well. Do you think someone is targeting cloud providers?
Could always just RTFM if SO goes down ️
I knew this will be here lol
Just replace Rails with Clojure.
As a huge fan of JetBrains products (I use WebStorm, PyCharm, DataGrip in roughly equal parts) I am considering:
https://www.jetbrains.com/space/
Anybody else got experience with that and/or TeamCity?
We use TeamCity and can't say I have any complaints. We utilize the tagging feature pretty heavily in our deployments and source all artifacts from TeamCity builds. The builds also post success or failure to Gerrit.
I feel compelled to repeat my earlier sarcasm:
If only there was some kind of distributed version control system. /s
I don't feel bad being an "old fogie" who demands local copies of my dependencies.
Github is as much or more a collaboration tool as a Git upstream. The Git part is easy to distribute. The collaboration tool, not so much.
And to anticipate the likely next argument - no, mailing lists are not a better tool for collaboration. They are distributed, sure. In so saying, I have exhausted the list of their virtues.
The "we deploy production directly from Github" pathology is the one I'm talking about. Not being able to "collaborate" for a brief outage is fine. Not being able to deploy code isn't.
10 replies →
Our nonprofit offers both privacy friendly cloud storage and gits with gitlab. We also offer gitlab hosting for private instances. https://git.stealthdrop.cloud and https://my.stealthdrop.cloud. Our service are free and we are 501c3 donations are deductible.