Incident Report: Railway Blocked by Google Cloud [resolved]

What if the reason for their stuff being shut down was a payment issue like an expired credit card or maxed credit account? Unless I missed it skim reading their post I don’t see any information anywhere about their communications with Google.

It's always possible to sue, but Google has good terms of service and lawyers - I'm 99% confident that a lawsuit would end up nowhere.

I can assure you that Google will be giving them significant commercial incentives as an apology for this behind the scene

I've been in AWS for almost twenty years at this point. It's been a long time since I've seen a global outage of the data plane on anything. The control plane, especially the US-east-1 services? Yes - but if you're off of east-1, your outages are measured in missile strikes, not botched deployments.

Perhaps you don't notice GCP outages because so few companies rely on them?

GCP never goes down because they banned all their customers.

GCP has had outages. From a quick search it looks like they had a global outage less than a year ago:

https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

AWS goes down catastrophically but are back up in minutes/hours most of the time (as long as they aren't down because Iran blew up their data center). That's obviously REALLY bad for certain industries, but I suspect for the vast majority of their customers it's not a big deal. We've been able to isolate the damage almost every time just by having AZ failover in place and avoiding us-east-1 where we can.

IIRC the Paris datacenter flood took down a whole “region” and some data was permanently unrecoverable.

>On the other hand i can’t remember when there was a serious outage on GCP

They had a really bad global outage a year ago. At least with AWS outages are contained to a single region.

You can't have 100% uptime. It's unfeasible, especially for a startup. You should be telling your customers that downtime might happen, sometimes for reasons beyond your control, and that if it does then you'll do your best to recover and to compensate them for the inconvenience. You should cultivate a relationship with your early customers that makes them feel bad for you when there's an outage rather than angry about how it impacts them. Maybe even go as far as firing the customers who give you a hard time over it. That way if your cloud provider falls over it's really annoying but not a big deal.

Your cloud provider blocking your business from running is far worse.

None of the AWS “outages” have impacted us. They have either been regional, in which case we stand down the region (we run multiple hot regions), or didn’t involve things we need to maintain operation.

I can’t imagine AWS ever doing such a cascading delete. I mean, they have made deletion protection a difficult thing to ignore even for individual resources.

Unfortunately, if everyone goes down people are understanding. If just _you_ go down, then its oddly less forgiveable.

How is blackhole-ing a customer not considered an outage?

I still remember the one where they nuked all the storage of I think an Australian insurance company I think, luckily the it department had done a multi cloud setup for backups

There was a pretty bad one last summer - their IAM system got a bad update and it broke almost all GCP services for an hour or so, since every authenticated API call reaches out to IAM.

It had lasting effects for us for a little over 3 hours.

You can read the parent post, right?

That’s an entirely different type of problem, and avoidable by just using us-east-2 (I still don’t understand why people default to us-east-1 unless they require some highly specific services).

If my cloud provider brings my startup down, it's my problem. If they bring all the startups down, that's their problem.

During my 5 years of my startup, we had only 1 outage due to AWS because we picked us-west-2 as the primary reason. If anyone starting a company and picks us-east-1 as the primary reason, they should be fired. There's absolutely no reason to be in that region.

And we all celebrate it since we can't do any work

hn.algolia.com gcp blocked

https://news.ycombinator.com/item?id=46731498 https://news.ycombinator.com/item?id=33737577

Do they?

The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

They better do. What is google doing?

i mean even google and aws are not without sin on this one. maybe wait for an RCA before punching someone who is currently down. theres a reason classy people do "hugops" when a competitor goes down, regardless of reputation.

Agreed, I'm very curious as to how this could happen.

But TheRegister did reach out to Google and they have not replied yet: https://www.theregister.com/off-prem/2026/05/20/google-cloud...

Yeah I’m planning a similar transition for my personal infrastructure. Railway is super easy to get started with, dashboard and logs features are nice, but I’ve just lost confidence in it.

https://blog.railway.com/p/incident-report-february-11-2026

I saw your youtube video, and while I am generally a fan of the small guy against big corps, this was a bit much with all the "I am so afraid to say something but I have to" talk.

So I will hold my judgement until this has been disected a bit more

According to the timeline the account was suspended for 18 minutes total. That is fast enough that it could have simply been a bug in a Google rollout or something that made something think it was suspended when it wasn’t really.

If it was actually suspended the yeah it’s weird not to get an email.

First time?

It's google, come on.

"UniSuper’s production Google Cloud VMware Engine (GCVE) private cloud was automatically deleted one year after it’s creation due to a misconfiguration in how it was created. When it was created, there was a bug in the creation script which passed a null value."

That's pretty amazing. Not due to a cascading failure from someone changing a config deep inside of a system that caused a bunch of unintended effects, just someone who messed up writing a shell script?

Creating stuff with 1yr (implicit) expiry by default is just a delayed footgun tbh

From personal experience, as a customer who once did something stupid: Google Cloud does soft deletes. But you need to reach out to support fast enough. And really, if you deleted something important and discovered it only the next day, and not within minutes, you're having a bigger issue that a soft delete won't solve.

It’s a good question. That said unless there are compliance or fallback concerns i would prefer a service that burns my data on departure.

Either mark-for-delete has the same impact as deleting in terms of shooting all the Cloud resources associated with the subscription, at which point the outage still happens but maybe the recovery is smoother or you've just delayed the inevitable by a week because no one will look at it unless there is actual impact.

> The instant cascading worldwide deletion upon closing or deleting a subscription sounds like a recipe for disaster.

I don't agree. What do you expect to happen when you explicitly delete your user account? Do you expect your systems to remain in operation for a week? That itself would be a major risk and liability, as your whole infrastructure would still be up even though you cut your access to it.

Also, isn't your whole infrastructure expected to be automatically deployed with IaC? The notable exception is data, which is already soft deleted and recoverable through customer support.

All in all, where do you expect the customer's responsibility to end and the cloud provider's to start? The shared responsibility model is covered by any intro course in no uncertain terms.

This is actually kind of validating. I work for a company that spends almost 1mm a year on GCP. We've never had an actual support contract with them because the numbers work out to, at a minimum, being 10% of our spend. We've yet to encounter a situation where we actually needed GCP support, so we've held off. In the moments where we'd like to get some support (mostly around datastore behavior) we've managed to work around it or figure it out ourselves. So it's good to know we haven't missed out on much. Beyond the offensive aspect of GCP offering no support if we aren't willing to cough up a non-trivial percentage of our spend, I'm pretty happy with it.

It's because they're measured on something, unsure which metric, but it's definitely not how helpful they are to you.

For what it's worth - I'm not sure what the criteria is (I assume we're "medium sized / not a big upsell opportunity"?) - our GCP rep quickly pushed us to switching to using a GCP reseller. They took over our billing so that we can pay via ACH, and provide both free first-line support/escalation and paid engagements for bigger projects; they don't charge a premium on top, apparently Google pays them for supporting us. Hasn't made much of a difference in how we operate, but at least we have a direct-ish line for issues when they come up.

That's exactly why I'm less pleased with GCP: to trust a CSP (or any service), I need to be assured that when (not if) things go wrong, I could escalate to a team that would have my back.

It doesn’t worry you enough that someday you could have a serious problem and they wouldn’t be able to help you?

What does blocked mean? Is there a different post that I am missing? There is shared infrastructure in GCP for networking (ex-googler here) and if only railway is affected, then it is not clear if it is only GCP or if there is something from Railway's perspective that needs to be addressed.

Hetzner is famously aggressive with their KYC (Know Your Customer) requirements, often locking new sign-ups and asking for photos of ID.

Damned if you do, damned if you don't.

Is it really a false positive if railway lets people run abusive services on GCP and then GCP consequently shuts them down?

I continue to receive phishing via AWS pretending to be Amazon. And not even the Unicode-lookalike shenanigans that my spam filter refuses for excessive mixed scripts, no; literally claiming to be Amazon as in: the company that operates the relay.

i wonder if DID or World (various ways of Proof of Human) can help solve this issue.

I'm not familiar with Railway, so this might not make any sense, but it's possible they were using their own hardware but managing it with Google accounts. It's not uncommon for a company's offsite human-to-human communications to fail when there's a Google outage or ban, so it's not unexpected to have the same interference with human-to-machine or machine-to-machine communications.

That’s strange, when I interviewed with the founder a few years ago he told me they were on AWS wanting to move to firecracker.

Yea, I mean, that's the whole MO of our platform and we failed at that. So yea, that's disappointing and more so for our customers.

I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto-modded out of our own account, hence we left our DB on CloudSQL.

It was never our intent to deceive people that we didn't own our own destiny with our business. The last GCP issue, we were assured that this scenario wouldn't happen (when we got auto-ratelimited, which was bad, but survivable) - but it seems like we have further work to do. Apologies.

Well, in the same token, is it smart to base your ENTIRE architecture on a single cloud architecture? Isn't that why some of us build in fallbacks for AWS-hosted services? I mean, their enitre platform, both public and private facing, is running on the same thing. One error, one problem, takes out the entire service.

They literally own their own data centers. That's whats surprising about this. They are lying to their customers when they say they operate their own data center because obviously they don't if everyone's apps are down with GCP blocking their account.

It surprises me there's not a manual review for $$$$ accounts. Speculation at this stage, but it's weird they would be put in the Recycle Bin like that.

god help them

We have a CSM, Head of Customer Support contact, and further contacts with GCP. Despite that, we still had this issue.

I became so conditioned to waiting hours(!) for DNS propagation that I'm always pleasantly surprised when it takes <5 min these days.

as with many things, we say we like decentralization but quietly vote for centralization

You’re giving Google far more credit than they’ve earned.

you can go on google cloud subreddit and watch horror stories

i actually built a good plan out of those horror stories for my companies.

I think it’s good people are making IaaS platforms, but have dealt with enough firefighter hero bullshit to have seen this coming a mile away. Uptime and redundancy are strongly correlated.

"Railway Blocked by Google Cloud"

If you don't happen to know that "Railway" is referring to a company, then you might reasonably read that as "a GCP outage caused issues in the train network somewhere".

99% of multi-cloud is fake though. True multi-cloud is incredibly rare.

Yeah, I'm not sure what to think here. We know Google is not the best at customer service and has automated account suspensions. But, what I'm curious about here is why this happened.

Railway hosts applications for customers. An uneducated guess for some possible reasons: 1) one of those customers hosted something they shouldn't have 2) railway had something spawn that took up too many resources 3) Or their account balance was too high 4) Or something...

But all of this probably culminates in someone needed to read an email that was missed.

Scaling a customer infrastructure setup like Railway is hard. This is one of the non-technical hard parts - how to make sure your account with your primary vendor is safe. But, I'm willing to wait to pass judgement here until more information is available. I'm sure the post-mortem will have lessons. I'd like to know more.

> via their contact method, aka the email they set it up with

If it's anything like AWS, that may be just one of hundreds of emails they send every day, most of which are just noise.

Honestly still insane to nuke a high-volume client's business after a single payment issue. There would be no reason for Google to believe that a single hiccup like that is evidence that they won't get paid and have to cut account access immediately.

Don't all major clouds do that by default? But at least they have additional protections you can configure, if you know about them.

Wish I could upvote this comment account more. Too many people look for something new and shiny when trusty ol tools are sitting right there. :)

Well having backups help, but I certainly can’t migrate my infra to rsync.net on moments’ notice (or ever since rsync.net does storage and nothing else) so my customers aren’t affected.

Inflated egress costs might make this prohibitively expensive, $80 per TB at GCP and AWS

I don't think that technology exists. Sorry.

I don't want to believe this, lol.

same

Low cost to entry, easy to get scale from the beginning if you need it. The large cloud providers throw free credit at startups to lock them in all the time. I had a short lived stint trying to get my own startup off the ground and it was really easy to get free compute from Google with no strings attached. This was many years ago now, but I would be surprised if it is any different.

I am with you entirely and would not have taken that route today, but it is really easy to see why people go that route.

A few years ago, when I was kinda active in the startup scene in my area, you have people selling access to cloud credits with penny-on-the-dollar price. The credits are given out liberally to big-corps, organization by AWS/GCP, through workshops, webinars, events. All in the hope of roping the departments into building MVPs, demos on AWS/GCP, but people also find a way to cheat on that system and make some quick bucks.

I know a startup of my acquaintances that have been running on AWS for 5 years straight without paying a single dollar to AWS. When the credits almost run out, they started to migrate their data over to another account with credit. That happened twice already.

It helps to have a portable, replicable IaC config. But also this is sustainable because they are a pretty small struggling shop. You will probably not be able to do this if you are trying to maintain more than 3 nines for an enterprise client.

Perhaps Railway does a bit more than what you think, they have some great functionality (I'm not affiliated with them). Check out [Features | Railway](https://railway.com/features) "PR Environments", they are incredible for the QA process

Oh absolutely... and many use architectures that have evolved out of the needs of really big companies and are not really a good fit for a startup. But I guess they want to be "ready for growth".

Railway deserves a lot of blame here. Deleting backups along with the database is a lot like not having backups. Moronic design choice.

16GiB Raspberry Pi 5s in my country are now going for ~$450USD, so I've gotta say that's out of reach for me now :(.

You’d think they wouldn’t have started with GCP. There are plenty of datacenters where you can buy racks and racks of servers, and talk to a human when something goes wrong, and even walk in and access your servers. That’s what I’d be using if I were to build a Rackspace today.

What do you mean by local small provider? A registrar on main street?

What the deuce are you blathering on about. An account got blocked, this has nothing to do with a domain.

And I’m talking about having disparate failovers that don’t rely on a single hosting provider. At that point, who cares what Google does to your cloud account… work with the hot failover and spin up another hot failover somewhere else.

MarkMonitor

Precisely. If you’re going to have a hot failover, it behoves you to have an entirely separate entity billing you for that hosting.

Honestly, I don’t know where the downvotes are coming from. Do people have no clue about service resiliency? I can understand if it’s a personal project or you haven’t yet scaled to paying customers, but anything at scale with serious money involved needs to be completely independent of the underlying hosting. It should remain up even if an entire provider goes titsup.