Comment by Animats
6 hours ago
"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover."
That's pretty clear. Google can no longer be trusted as a B2B service provider.
Meta is no different. I know a company that had their OAuth app on Meta rendered completely unusable just because one of their employees (a dev) had their personal Facebook account banned by Meta for no reason. They tried to escalate it multiple times but got nowhere, lol. Meta is even worse because accounts need to be 'personal'; if you have a Business Manager, the users added to it are all tied to their personal Meta/Facebook accounts. This is ludicrous.
To me, building any business with dependencies on Meta is just a bad business plan.
Meta and Google B2B are both horrible. Their ad account bans are constant, and they have no real escalation process to get help. These companies are monopolies that should treat businesses more seriously, especially in these situations.
Yeah, people loose their business because a kid is logged in on their iPad, gets their google account suspended, and google knows it's the same household as the parent, and everything gets shut down
Can't find this now but google did at least once disable company's accounts after dev got their account suspended.
And as we know from the recent Gemini ban wave, you can get suspended just because.
Everyone needs a defensible root of trust, this goes all the way down to the registrar you use for your domain.
> google knows it's the same household as the parent,
Nearly all these linkages are due to people sharing recovery email addresses and phone numbers. Don't do that.
3 replies →
[flagged]
The is in context of B2B, which meta has a huge ecosystem and often rips away a companies revenue for hidden reasons
1 reply →
Seems relevant to me as it is still a service that their company relied on.
1 reply →
They're a popular SSO provider.
A huge number of small businesses have no Internet presence beyond their Facebook and Insta pages, so … yes they are extremely relevant to a discussion about the risk to small business of flaky hyperscalers.
More businesses need to hear this message. Google has proven time and time again they cannot be trusted as a service provider, exactly because of this problem.
They trust them enough to still give them money, just goes to show how entrenched big tech is and why they need to be broken up into dozens of pieces.
In the meantime start by migrating away from them for anything serious.
There is a history going back many years of Google suspending or terminating accounts with no explanation, often having to backtrack after users published their frustration and the incident went viral.
Google has always acted as if they have no obligations whatsoever to their paying customers.
They have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.
> "Cloud Providers don't suspend entire accounts for no reason."
Maybe I'm getting old but here[1] is a HN comment from 17 years ago complaining about Google banning accounts "by mistake" and having no recourse but to post on HN and hope Matt Cutts sees it and helps, and saying "there are literally 1000s of such stories for many years all over the blogoshphere and forums" which is something I remember from HN of years ago.
[1] https://news.ycombinator.com/item?id=791004
And unfortunately nothing has changed since then regarding this.
The cloud provider in question - GCP - who also deleted a 125 billion dollar company's entire account on accident?
What company?
3 replies →
> Cloud Providers don't suspend entire accounts for no reason.
You're joking, right?
LOL, did you woke up from the hibernation?
This is Google we're talking about. This absolutely happened many times in the past and will happen again.
Google has suspended entire accounts countless times for absolutely no reason.
Unfortunately the cloud providers also rarely if ever tell you the reason.
Not defending them. but wouldn't it be a legal nightmare if they did?
My guess would be the credit card expired....
If it were something out of Railways hands, I think they would say something like "We have not yet identified the reason for the suspension, and are awaiting a response from Google".
2 replies →
FTA:
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
This might be 100% of what google told them.
Somebody else's computer
Never could. Google might block your entire company because one of your workers did something nasty on their personal account, and their ban hammer is mighty and blocks all related accounts to the Nth degree
I've been wondering if I should be interrogating my friends before allowing them access to my wifi. "Have you or any of your family members ever been banned by Google?"
Hasn't every cloud provider had issues? Is the enshitification of servces really isolated to Google, or are we all doomed.
Banning accounts for no stated reason is kind of a Google speciality. They have a long well documented history of this sort of thing.
Railway has an overwhelming incentive to pin the blame on Google. This report doesn't answer why Google suspended Railway's account.
I'd wait for more details before adjudicating.
In principle, I agree with you.
In practice, Google has earned the way my priors are ready to believe it's 100% their fault with mighty and sustained effort. Or lack thereof, depending on your point of view.
That would be approximately 6365262822 time Google suspended someone for no good reason.
So no, Google doesn't get the benefit of the doubt.
They said it was automated and affected a bunch of other customers, which gives at least some hint.
And in general Google lost any immediate benefit of the doubt status many years ago. Many such stories.
To quote the article:
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
I'm not sure that's the lesson to learn from this outage. Hell Google resolved the problem in 7 minutes which is as good as you could hope for.
The resulting action should be you have proper disaster recovery, failover, etc.
Not sure I would trust these folks if this is the conclusion they are coming to from this experience. Any cloud provider can/will do this to you.
Google restored access but did not resolve the problem. VM’s were still shut down.
Google resolved the problem in seven minutes for a billion dollar company. Good luck if you are a nobody.
The best you could hope for is that if there is something fishy with your account, you are contacted by Google to address it.
Railway don't have a great reputation for building scalable systems (effects of vibe coding?). It's worth waiting for Google's response before jumping to conclusions. They can move to Azure/AWS/own datacenter, but there's a good chance this will repeat in a few months.
Sure, if this was one off isolated incident people would have agreed with you. But it's not. Even Google personal accounts have been used to ban their other ones including ones spending thousands of dollars on ads or GCP or any other paid google service, which is ridiculous.
Their reputation is fine, and their uptake is due in part to their handling of scaling.
If you're picking them instead of the underlying cloud provider, but you want all the knows and dials the underlying provider has, you've made the wrong choice.
There is always one bootlicker, fresh 1 day account no less.
Been a passive reader here at HN for too long, finally registered today. Instead of viewing this incident objectively, you choose to insult me (?).
I know multiple startup founders personally (2 of them are in the current YC batch), and the sheer callousness with which they look at infra, especially from security/scalability/reliability angle is shocking.
I'll personally reserve judgement against GCP (replace with AWS/Azure/OCI/whatever) until we know more.
Then let me be the not day-one account to say Railway is utterly bearing some responsibility here.
"However, in this ring, there was still a hard dependency on workload discoverability being tied to the network control plane API that was hosted on the machines running in Google Cloud."
They've gotta be joking me that they deliberately left something so critical under the control of any other entity than themselves. That demonstrates a lack of critical planning and a lack looking at their configuration from a first-principles approach.
1 reply →