It looks like that it is a central service @ Google called Chemist that is down.
"Chemist checks the project status, activation status, abuse status, billing status, service status, location restrictions, VPC Service Controls, SuperQuota, and other policies."
-> This would totally explain the error messages "visibility check (of the API) failed" and "cannot load policy" and the wide amount of services affected.
There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.
At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.".
So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.
100% agree... I even thought "ok maybe I'll clean up the backlog while I wait" but I'm so used to even using AI to clean up my JIRA backlog (using the Atlassian MCP), so even that feels weird to click into each ticket, just the way I used to do it TWO MONTHS AGO.
This is a good wake-up call on how easily (and quickly) we can all become pretty dependent on these tools.
Update - We are seeing a number of services suffer intermittent failures. We are continuing to investigate this and we will update this list as we assess the impact on a per-service level.
Impacted services:
Access
WARP
Durable Objects (SQLite backed Durable Objects only)
Workers KV
Realtime
Workers AI
Stream
Parts of the Cloudflare dashboard
Jun 12, 2025 - 18:48 UTC
Some day Cloudflare will depend on GCP and GCP will depend on Cloudflare and AWS will rely on one of the two being online and Cloudflare will also depend on AWS and the internet will go down and no one will know how to restart it
Yeah. This service was presenting charts likely probed from inside GCP. I was on a call with a Google rep, someone pointed out that "AWS is also down" and I foolishly said something about "possible BGP attack" out of spite, before checking AWS availability myself. Shame on me.
I love this kind of fake news. It's like that scene from Scary Movie (can't remember which one) in which someone says "I heard the japs took out one in Kikoman" :')
Why even have a status page? Someone reported that their org of >100,000 users can't use Google Meet. If corps aren't going to update their status page, might as well just not have one.
Edit: The GCP status page got updated <1 minute after I posted this, showing affected services are Cloud Data Fusion, Cloud Memorystore, Cloud Shell, Cloud Workstations, Google Cloud Bigtable, Google Cloud Console, Google Cloud Dataproc, Google Cloud Storage, Identity and Access Management, Identity Platform, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Vertex AI Search
There's no situation where the corporation controls the status page where you can trust the status page to have accurate information. None. The incentives will never be aligned in this regard. It's just too tempting and easy for the corp to control the narrative when they maintain their own status page.
The only accurate status pages are provided by third party service checkers.
I have zero faith in status pages. It's easier and more reliable to just check twitter.
Heroku was down for _hours_ the other day before there was any mention of an incident - meanwhile there were hundreds of comments across twitter, hn, reddit etc.
Why can't companies be honest with being down. It helps us all out so we don't spend an hour internalizing.
We are truly in gods hands.
$ prod
Fetching cluster endpoint and auth data.
ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=503, message=Visibility check was unavailable. Please retry the request and contact support if the problem persists
Because a lot of the time, not everyone is impacted, as the systems are designed to contain the "blast radius" of failures using techniques such as cellular architecture and [shuffle sharding](https://aws.amazon.com/builders-library/workload-isolation-u...). So sometimes a service is completely down for some customers and fully unaffected for other customers.
if half the internet is down, which it apparently is, it's usually not the service in question, but some backbone service like cloudflare. And as internal health monitoring doesn't route to the outside through the backbone to get back in, it won't pick it up. Which is good in some sense, as it means that we can see if it's on the path TO the service or the service itself.
Yeah, my company of hundreds of people working remotely are having 90%+ failures connecting to Google Meetings - joining a meeting just results in a 504.
Whichever product person is in charge of the status page should be ashamed
How could you possibly trust them with your critical workloads? They don't even tell you whether or not their services work (despite obviously knowing)
My guess is whatever system downdetector uses to "detect downtime" relies on either GCP or Cloudflare (also having issues at the moment: https://www.cloudflarestatus.com/)
Aren't some of these sites partially based on hits (because of the assumption that if enough people are suddenly googling "Is youtube down", then youtube must be having some sort of issue.
I could see a big outage like this causing people to google "Is AWS down?"
Yes Firebase auth is down and affecting many apps, on Discord and Slack groups tons of others are corroborating. A bit disappointing that there is no post on the status page for nearly 30 mins:
https://status.firebase.google.com/
> Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information
Does anyone know of a good dashboard to check for such BGP routing anomalies as (apparently) this one? I am currently digging around https://radar.cloudflare.com/routing but it doesn't show which routes were actually leaked.
I would love if anyone has any good tool recommendations!
Cloudflare runs all their own bare metal servers. Seems odd that they would be impacted by Google cloud. Same can be said for all the other issues on downdetector. This points to a broad issue at the core internet which could certainly be related to BGP.
Initially attributed the unresponsiveness of `npm install` to npm (the CLI tool) in general. Tried using bun to install dependencies, saw the same result -- but with actual logs instead of a vague spinner -- and decided to check Hacker News.
Getting 504 errors on anything from registry.npmjs.org that isn't cached on my machine.
Interesting how I landed here. I was having trouble with Nest. Then I went to Down Detector. I noticed many sites having a simultaneous uptick. Then I came to HN, and found this link at the top of the front page.
If Google Chat is down per https://www.google.com/appsstatus/dashboard/, the ability for Google engineers to communicate among themselves impaired, despite SREs having IRC as a backup.
Yep. Self-reporting status pages are pretty near worthless. At my former large company (not FAANG), we weren't allowed to update the status page until we got VP approval, which also required approval from both PR and Legal. It would take a lot more time and effort to get those approvals than to just fix the problem and move on.
SLA contracts, clawbacks, and performance obligations make these pages a bit of a minefield for CSPs. When I was at a top-tier CSP, we had the status page that was public, one that was for a trusted tier of customers, one built for a customer-by-customer basis, and one for internal engineering.
Sorry, after decades of being hard wired, I just installed a PCIe Wifi6 card on my desktop. Internet took a dive the second I got it connected. Must have done something wrong.
Status pages at cloud providers aren't usually based in reality -- usually requires VP level political games to actually get them changed especially for serious outages.
Update - Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency.
Jun 12, 2025 - 19:57 UTC
I'm sure it's not entirely impossible, but sounds backwards to me. Sure - a lot of the internet relies on Cloudflare, but I'd be very surprised if GCP had a direct dependency on Cloudflare, for a lot of reasons. Maybe I misunderstood your comment?
It’s not a euphemism - every outage, including the 99.9% that don’t end up on HN gets a postmortem document written about it, which is almost always a fascinating discussion of the technical, cultural and organisational situation that led to an unexpected bad thing happening.
Even a few years ago senior management knew to stay the fuck out except for asking for more info.
Let's say a typical base service (network attached RAM or whatever) has 99.99% reliability.
If you have a dependency on 100 of those, you're suddenly closer to 99% reliability.
So you switch to higher-level dependencies, and only have 10 dependencies, for a 99.9% reliability.
But! It turns out, those dependencies each have dependencies, so they're really already more like 99.9% at best, and you're back at 99% reliability.
"good enough" is, indeed, just good enough to make it not worthwhile to rip out all the upstreams and roll your own everything from scratch, because the cost of the occasional outages is much lower than the cost of reinventing every single wheel, nut, bolt, axle, bearing, and grease formulation.
Looks like I'm about to start learning which of my time-killing websites are hosted on GCP - The Ringer is down, and since Spotify owns them and is a major GCP customer, it looks like they've been hit by this. CRAZY that the GCP status page is still green.
Just our bi-yearly reminder of our over reliance on cloud providers for literally everything. Can't say there's an answer beyond trying to build more independent tech but we know how that goes.
Yet migration to the cloud continues, driven by people arguing that doing it yourself is too complicated and expensive. Let’s see how long until one outage takes down the global economy for multiple days or weeks.
"The cloud" isn't the problem nor is migration to it. The problem is single points of failure in a vast, super-connected, global network.
To demonstrate my point, say someone like cloudflare opted to get "off cloud" and run their own datacenters. Half the web would still go down if they had some issue in their datacenters. If anything, the economy of scale and resiliency of a huge cloud network is far beyond what any single operator can ensure for their own service. If this wasn't the case, cloud services couldn't be as profitable as they are.
It isn't the panacea people seem to think it is. One critical service going down, regardless of whether it's cloud hosted or not, has ripple effects in the broader network.
Hilariously, I did not know about any outages today during the workday because we discourage cloud service usage and nobody complained about anything breaking. :)
First, check that nobody else in your family is making a call on the phone line that your modem is connected to, then make sure to disable your Internet Explorer add-ons before trying again.
Was just about to do a demo, but Google Meet was down. Tried to use Jitsi as a fallback, but couldn't log in because Firebase was down too. Ended up using a Slack Huddle, lol.
For us Cloud SQL instances are toast but App Engine Standard instances are still serving requests. Google Cloud console is borked too, mostly just erroring out.
some core GCP cloud services are down. might be a good time for GCP dependent people to go for a walk, do some stretches, and check back in a couple hours.
Haha, I don't ordinarily spend a lot of time in the Google Cloud Console but just now I was debugging a squirrely OAuth issue with reCAPTCHA failing to refresh several days running. I'm getting this weird page error, and I think, "Is this an issue with my organization? [futz futz futz] Hey wait is GCP actually down?" And it turns out to be the top discussion on HN. XD
Can't wait to see how charts are going to look like here on the project we have developed for Maintel https://variable.io/maintel-digital-landscape/. It shows availability across multiple services as a landscape. Expecting to see a lot of spikes tomorrow.
A contact in google mentioned to me that some bad update to Google Cloud Storage service has caused some cascading issues affecting multiple GCP services.
I doubt gcloud would be affected by an aws-specific cni. Unless maybe enough AWS users have a GCP backup environment that they flipped on all at once, but it seems unlikely
When Google said GCP is "down", did it affect entire availability zones within a region? For people who designed redundant infrastructure, did your backup AZs/regions keep your systems online?
The outage was global. For my team specifically, a global Identity and Access Management outage meant that our internal service accounts could not refresh their short-lived access tokens and so different parts of our infrastructure began to fail over the course of an hour or so, regardless of what region or zone they were in. Services were up, but they could not access critical GCP services because of auth-related issues which resulted in internal service errors for us.
To give an example, our web servers connect to our GCP CloudSQL database via a Cloud SQL Auth Proxy (there's also a connection pooler in between but that also stayed up). The connection to the proxy was always available, but the proxy wasn't able to renew auth tokens it uses to tunnel to the database, regardless of where the webserver or database happened to be located. To mitigate this in the future we're planning to stop using the auth proxy and connect directly via mutual TLS but now it means we have to manage TLS certificates.
reCAPTCHA affected? I couldn't log into my local utilities website due to a reCAPTCHA error. Downdetector agrees, but I interpret that site as dubious.
I'm on a meet, in cal, editing a dozen docs, in GCP, pushing commits and launching containers; it's not clear yet what exactly is going on but it's certainly intermittent and sparse, at least so far
An hour later and everything is a mess in central-1. They seemed to jump the gun on that one. Doesn't matter if some dinky service like "AutoML Vision" is working, if GCS isn't, then they shouldn't post an optimistic message.
"Firebase Data Connect unavailable due to a known Google Cloud global outage"
While the Google Cloud status page https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it.
Edit to add: within 10 minutes of this post Google updated their status page. More curiously the Firebase page I linked to has been edited to remove mention of Google Cloud in the status and now says "Firebase Data Connect is currently experiencing a service disruption. Please check back for status. ".
IIRC status pages drive customer compensation for downtime. Updating it is basically signing the check for their biggest customers, in most similar companies you need a very senior executive to approve the update
On the other side of this, Firebase probably doesn't have money at stake making the update
Nah, its just some client side caching / JS stuff. Clicking the big refresh button fixed it for me, 15 minutes before OP noted it.
(n.b. as much as Google in aggregate is evil, they're smart evil. You can't avoid execs approving every outage because checks without some paper trail, and execs don't want to approve every outage, you'd have to rely on too many engineers and sales people, even as ex-employees, to keep it a secret. disclaimer: xoogler)
(EDIT: for posterity, we're discussing a "overall status" thing with a huge refresh button, right above a huge table chockful of orange triangles that indicate "One or more regions affected" - even when the "overall status" was green, the table was still full of orange and visible immediately underneath. My point being, you gotta suppose a wholeeee bunch of stuff to get to the point there was ever info suppressed, much less suppressed intentionally to avoid cutting checks)
Something must be preventing them updating the status page at this point. Of course they could still deem it not enough, but just from my limited tests, docker, buf, etc (it may not be GCP that is down, but it is quite the coincidence). are outright down. I'd wager that this is much more widespread.
I'm actually on a bridge call with Google Cloud, we're a large customer -- I just learned today that their status page is not automated, instead someone actually manually updates it!
This extra funny that GCP status page even includes a “last updated” time, which is exactly built to convey possible failure to update in cases like this
No major incident as of “ Last updated time: 12 Jun 2025, 11:48 PDT”
The cloud enables you to scale. It allows us to distribute systems across multiple regions and data centers. Seems that this is true for outages as well.
The PHP application I wrote as a student running on a single self-hosted server had a higher uptime than any of the cloud providers or redundant system I have seen so far. If you don’t need the cloud for scalability, do it yourself and save yourself the trouble and money. Most companies would be better off investing into some IT staff instead of giving away their systems in the hands of some proprietary and insanely complex cloud environment. You are becoming dependent on someone you don’t know, have no control over and can’t talk with directly. Also the single point of failure is just shifting: from your system to whatever system is managing the cloud. Guess one advantage is that you can shift the blame to someone else…
It looks like that it is a central service @ Google called Chemist that is down.
"Chemist checks the project status, activation status, abuse status, billing status, service status, location restrictions, VPC Service Controls, SuperQuota, and other policies."
-> This would totally explain the error messages "visibility check (of the API) failed" and "cannot load policy" and the wide amount of services affected.
cf. https://cloud.google.com/service-infrastructure/docs/service...
EDIT: Google says "(Google Cloud) is down due to Identity and Access Management Service Issue"
There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.
Absolutely possible. Though there is something curious:
https://www.cloudflarestatus.com/
At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.".
So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.
36 replies →
perhaps the person who maintains Chemist took the buyout
https://www.businessinsider.com/google-return-office-buyouts...
I use Expo intermediation for notifications, but with this Google context, I imagine that FCM is also suffering, is that possible?
Very likely. Firebase Auth is down for sure (though unreported yet), so most likely FCM too
2 replies →
Getting a lot of errors for Claude Sonnet 4 (Cursor) and Gemini Pro.
Nooooo I'm going to have to use my brain again and write 100% of my code like a caveman from December 2024.
Same here. Getting this in AI Studio: Failed to generate content: user has exceeded quota. Please try again later.
[flagged]
3 replies →
I was in the middle of testing Cloud Storage file uploads, so I guess this is a good time to go for a walk.
A good excuse for adding error handling, which otherwise is often overlooked, heh.
Cursor throwing some errors for me in Auto Agent mode too.
Devs before June 12, 2025: "Ai? Pfft, hallucination central. They'll never replace me!"
Devs during June 12, 2025 GCP outage: "What, no AI?! Do you think I'm a slave?!"
100% agree... I even thought "ok maybe I'll clean up the backlog while I wait" but I'm so used to even using AI to clean up my JIRA backlog (using the Atlassian MCP), so even that feels weird to click into each ticket, just the way I used to do it TWO MONTHS AGO.
This is a good wake-up call on how easily (and quickly) we can all become pretty dependent on these tools.
1 reply →
It appears like "Devs" is not a homogeneous mass.
Goomba fallacy
So true
openrouter.ai is down for me
switch to auto mode and it should still work!
GPT is working in agent mode, which kind of confirms that claude is hosted on google and GPT probably on MSFT servers / self hosted.
3 replies →
lmao i refuse to write code by hand anymore too. WHAT IS THIS
I chose sepuku.
Apple’s local models looking better each day :’)
Google's local models as well (Gemini Nano/Gemma 3n)
2 replies →
Cloudflare is down too. From https://www.cloudflarestatus.com:
Update - We are seeing a number of services suffer intermittent failures. We are continuing to investigate this and we will update this list as we assess the impact on a per-service level.
Impacted services: Access WARP Durable Objects (SQLite backed Durable Objects only) Workers KV Realtime Workers AI Stream Parts of the Cloudflare dashboard Jun 12, 2025 - 18:48 UTC
Edit: https://news.ycombinator.com/item?id=44261064
Seems like a major wtf if Cloudflare is using GCP as a key dependency.
Some day Cloudflare will depend on GCP and GCP will depend on Cloudflare and AWS will rely on one of the two being online and Cloudflare will also depend on AWS and the internet will go down and no one will know how to restart it
7 replies →
Broken link? EDIT: Weird, definitely was just empty
Should work, but its also on the front page.
Everything appears to be down as of 18:43 UTC... https://downdetector.com/
Yeah. This service was presenting charts likely probed from inside GCP. I was on a call with a Google rep, someone pointed out that "AWS is also down" and I foolishly said something about "possible BGP attack" out of spite, before checking AWS availability myself. Shame on me.
Didn't have the feeling of a BGP issue, most services I was working with were reasonably quickly returning failures, as opposed to lingering death.
I love this kind of fake news. It's like that scene from Scary Movie (can't remember which one) in which someone says "I heard the japs took out one in Kikoman" :')
Wait, it's all Google?
Google was the first to report probably.
all cloud
"always has been"
Well that's interesting. I wouldn't expect AWS or Microsoft 365 to be affected by a Google outage.
Who said it's a Google outage?
3 replies →
Perhaps their detection logic is running on Google cloud /s
I believe Downdetector displays user reports.
2 replies →
Downdetector in incidents like this is 100% misinformation.
Why
3 replies →
The status page is green, but there are outages reported: https://downdetector.com/status/google-cloud/
Why even have a status page? Someone reported that their org of >100,000 users can't use Google Meet. If corps aren't going to update their status page, might as well just not have one.
https://www.google.com/appsstatus/dashboard/
https://status.cloud.google.com/index.html
Edit: The GCP status page got updated <1 minute after I posted this, showing affected services are Cloud Data Fusion, Cloud Memorystore, Cloud Shell, Cloud Workstations, Google Cloud Bigtable, Google Cloud Console, Google Cloud Dataproc, Google Cloud Storage, Identity and Access Management, Identity Platform, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Vertex AI Search
There's no situation where the corporation controls the status page where you can trust the status page to have accurate information. None. The incentives will never be aligned in this regard. It's just too tempting and easy for the corp to control the narrative when they maintain their own status page.
The only accurate status pages are provided by third party service checkers.
3 replies →
Who gets a promotion from a working status board?
I have zero faith in status pages. It's easier and more reliable to just check twitter.
Heroku was down for _hours_ the other day before there was any mention of an incident - meanwhile there were hundreds of comments across twitter, hn, reddit etc.
1 reply →
> might as well just not have one
This is my position.
Here's the incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
It was nearly an hour into our company's internal incident channel on this for GCP to finally declare that yes, in fact, things on fire.
… I get that PR-types probably want to massage the message, but going radio dark is not good PR.
Why can't companies be honest with being down. It helps us all out so we don't spend an hour internalizing.
We are truly in gods hands.
$ prod
Fetching cluster endpoint and auth data. ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=503, message=Visibility check was unavailable. Please retry the request and contact support if the problem persists
Because they have unrealistic targets so they make up fake uptime numbers. 99.999% would mean not even having an hour of downtime in 10 years.
I remember reddit being down for like a whole day or so and they claimed 99.5% in that month.
4 replies →
Because a lot of the time, not everyone is impacted, as the systems are designed to contain the "blast radius" of failures using techniques such as cellular architecture and [shuffle sharding](https://aws.amazon.com/builders-library/workload-isolation-u...). So sometimes a service is completely down for some customers and fully unaffected for other customers.
10 replies →
Because there are contracts related to uptime :)
5 replies →
if half the internet is down, which it apparently is, it's usually not the service in question, but some backbone service like cloudflare. And as internal health monitoring doesn't route to the outside through the backbone to get back in, it won't pick it up. Which is good in some sense, as it means that we can see if it's on the path TO the service or the service itself.
> Why can't companies be honest with being down
SLA agreements.
2 replies →
The program that updates the status page is hosted on Google Cloud.
13 replies →
Nobody gets a promotion, that's why.
Please, won't somebody think of the KPIs.
Yeah, my company of hundreds of people working remotely are having 90%+ failures connecting to Google Meetings - joining a meeting just results in a 504.
It's updated now, shows the impact to console, dataproc, GCS, IAM and Identity Platform: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
Whichever product person is in charge of the status page should be ashamed
How could you possibly trust them with your critical workloads? They don't even tell you whether or not their services work (despite obviously knowing)
[dead]
AWS is fine: https://health.aws.amazon.com/health/status
My guess is whatever system downdetector uses to "detect downtime" relies on either GCP or Cloudflare (also having issues at the moment: https://www.cloudflarestatus.com/)
So’s Azure? https://downdetector.com/status/windows-azure/
This is where we get to learn about the one common system all of our “distributed cloud” systems rely on, isn’t it?
1 reply →
Aren't some of these sites partially based on hits (because of the assumption that if enough people are suddenly googling "Is youtube down", then youtube must be having some sort of issue.
I could see a big outage like this causing people to google "Is AWS down?"
Almost everything on the downdetector home page is listed as having downtime...
1 reply →
wtf is going on
It's the entire internet. Check oracle cloud, etc etc. The ENTIRE INTERNET.
5 replies →
What's crazy is that RCS messaging is down as a result of this outage. It shows how poorly the technology or infrastructure was designed.
Isn't RCS basically just instant messaging? I don't know why it's surprising that it would be down.
I'm not sure any single company could have an outage that would take out SMS globally, but RCS is presumably more centralized.
5 replies →
That explains why I couldn't get the photo of my parents dog today.
should have used Erlang
Oh my god is that why my RCS chats were failing earlier?!?!
Yes Firebase auth is down and affecting many apps, on Discord and Slack groups tons of others are corroborating. A bit disappointing that there is no post on the status page for nearly 30 mins: https://status.firebase.google.com/
It just updated. Maybe affected by their own outage!
Just proves how shady the status page and sla stuff is
7 replies →
Cloudflare Outage also just updated
> Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information
Is GCP the third party?
Does anyone know of a good dashboard to check for such BGP routing anomalies as (apparently) this one? I am currently digging around https://radar.cloudflare.com/routing but it doesn't show which routes were actually leaked.
I would love if anyone has any good tool recommendations!
I don't know if I've seen CF Radar before. That's pretty cool!
Here are some others, although some seem to be experiencing issues due to the current outage I can only presume.
- https://atlas.ripe.net/probes/public
- https://www.ihr.live/en/global-report
- https://www.ihr.live/en/network
- https://bgp.he.net/
- https://ioda.inetintel.cc.gatech.edu/dashboard/asn
My default go-to: https://bgp.tools/
Why would you think this outage is (internet) BGP related?
Cloudflare runs all their own bare metal servers. Seems odd that they would be impacted by Google cloud. Same can be said for all the other issues on downdetector. This points to a broad issue at the core internet which could certainly be related to BGP.
2 replies →
I am a newb at this too, but is it "normal" for the "Announced IP Address Space" section to have that large jump from addresses like that?
go https://status.gcp.databricks.com/
BGP attack?
thank god hn is hosted on a single bare metal server, free of all this bloat.
I can recall plenty of times HN has been down.
it has already been facing some issues in the past couple weeks. e.g. the "front" page was broken randomly for past dates (like https://news.ycombinator.com/front?day=2025-05-30 or https://news.ycombinator.com/front?p=2)
as per their api documentation [1], it might be linked to firebase, which might explain this?
[1] https://github.com/HackerNews/API
1 reply →
Smells like BGP since there are services people claim have nothing to do with GCP being affected. OpenRouter is down, Lovable is down, etc.
AWS seems fine though. My bet is Cloudflare.
AWS and Azure both had outages.
3 replies →
perhaps Lovable uses GCP somewhere in their stack?
npm as well
Initially attributed the unresponsiveness of `npm install` to npm (the CLI tool) in general. Tried using bun to install dependencies, saw the same result -- but with actual logs instead of a vague spinner -- and decided to check Hacker News.
Getting 504 errors on anything from registry.npmjs.org that isn't cached on my machine.
1 reply →
Interesting how I landed here. I was having trouble with Nest. Then I went to Down Detector. I noticed many sites having a simultaneous uptick. Then I came to HN, and found this link at the top of the front page.
Same here with npm
https://status.npmjs.org/incidents/dn5mcp85737y
I usually just go here first.
If Google Chat is down per https://www.google.com/appsstatus/dashboard/, the ability for Google engineers to communicate among themselves impaired, despite SREs having IRC as a backup.
TIL Google chat hasn't been killed yet
They have irc services internally (or at least did when I was there 10-ish years ago).
Google Chat wasn't down for me throughout the entire incident.
it at least used to be standard and fairly well known practice for non-sres to use the irc bridge.
the much more disastrous situation would have been the irm fallback.
Someone actually uses Google Chat...?
Google has a chat product?
it's the best
4 replies →
Almost everyone inside Google
> No major incidents
… Proceeds to show worldwide degraded service level alerts.
Yep. Self-reporting status pages are pretty near worthless. At my former large company (not FAANG), we weren't allowed to update the status page until we got VP approval, which also required approval from both PR and Legal. It would take a lot more time and effort to get those approvals than to just fix the problem and move on.
SLA contracts, clawbacks, and performance obligations make these pages a bit of a minefield for CSPs. When I was at a top-tier CSP, we had the status page that was public, one that was for a trusted tier of customers, one built for a customer-by-customer basis, and one for internal engineering.
1 reply →
Sorry, after decades of being hard wired, I just installed a PCIe Wifi6 card on my desktop. Internet took a dive the second I got it connected. Must have done something wrong.
This is at least why Claude is dead: https://status.anthropic.com/incidents/kn7mvrgb0c8m
Also spotify isn't working for me so I assume that's also related.
These are my most important productivity resources! Sad!
Incident report published: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
Status page is showing green because GCP admins can't login to change it ;)
Status pages at cloud providers aren't usually based in reality -- usually requires VP level political games to actually get them changed especially for serious outages.
Look like affect to Cloudflare as well [1]
1: https://www.cloudflarestatus.com/
Would be comedy if one of the progenitors of this took Sundar’s buyout offer yesterday and let the world burn today.
Kinda funny that the top post on HN titled "GCP Outage" links to the Google Cloud status page which shows...no outage.
They have updated the status page finally https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
Does anyone know if it's region-specific? We're experiencing it and are in us-west-1.
Us-central-1 as well
Can confirm us-east1 (and possibly us-south1) are having VPC host reachability problems.
it's due to IAM and global
Frankfurt seems to be down as well
europe (netherlands) region as well
us-east-1 too
south korea as well
https://www.cloudflarestatus.com/ is showing outage, which cause google gcp outage, claude outage, firbase outage https://status.firebase.google.com/
How would Cloudflare's outage cause a GCP outage?
I'm sure it's not entirely impossible, but sounds backwards to me. Sure - a lot of the internet relies on Cloudflare, but I'd be very surprised if GCP had a direct dependency on Cloudflare, for a lot of reasons. Maybe I misunderstood your comment?
This appears to be continuing to cascade over an hour later... wow... more and more services mentioned as completely down on the outage page.
Kind of nice to not be glued to AI chat prompts for a while to be honest.
Everyone is down. Cloudflare has problems too. All auth providers broken.
Someone must have checked in AI Generated code :-)
Super duper frustrating having the status page being green. Why can't Google do this properly?
Those responsible have been sacked.
Those responsible for sacking the people who have just been sacked, have been sacked.
Wish there existed a decentralized network connecting computers around the world
Crazy, they could call it the "internet" or something like that... kind of rolls off the tongue.
https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
> Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue
IAM issue huh. The post-mortem should be interesting at least.
Ha. With all this soviet style euphemism I rather read the onion instead.
It’s not a euphemism - every outage, including the 99.9% that don’t end up on HN gets a postmortem document written about it, which is almost always a fascinating discussion of the technical, cultural and organisational situation that led to an unexpected bad thing happening.
Even a few years ago senior management knew to stay the fuck out except for asking for more info.
Google Maps not loading, thought it was my 4g, go to see if my connection works by loading Hacker News, GCP Outage XD
Cloudflare speedtest is down too, I assume because of this?
Appears to be a separate incident: https://news.ycombinator.com/item?id=44261064
Two big cloud provider outages at the same time? Has to be related surely.
Our site depends on Workers and KV and it's very broken right now. Can't login to the Cloudflare Dashboard either.
Works for me in Portland on Quantum Fiber.
Let's say a typical base service (network attached RAM or whatever) has 99.99% reliability. If you have a dependency on 100 of those, you're suddenly closer to 99% reliability. So you switch to higher-level dependencies, and only have 10 dependencies, for a 99.9% reliability. But! It turns out, those dependencies each have dependencies, so they're really already more like 99.9% at best, and you're back at 99% reliability.
"good enough" is, indeed, just good enough to make it not worthwhile to rip out all the upstreams and roll your own everything from scratch, because the cost of the occasional outages is much lower than the cost of reinventing every single wheel, nut, bolt, axle, bearing, and grease formulation.
Looks like I'm about to start learning which of my time-killing websites are hosted on GCP - The Ringer is down, and since Spotify owns them and is a major GCP customer, it looks like they've been hit by this. CRAZY that the GCP status page is still green.
One of these days in which the young engineers learn the concept of 'counterparty risk'.
I wonder what the damage ($) for having a good portion of the internet down for an hour or two ;)
So frustrating, but here's a link to track status of this outage: https://status.anthropic.com/incidents/kn7mvrgb0c8m
Actual incident link posted: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
Supabase is also down
Yes my project on Supabase is down as well.
When you deploy code generated by Gemini :D
Just our bi-yearly reminder of our over reliance on cloud providers for literally everything. Can't say there's an answer beyond trying to build more independent tech but we know how that goes.
Yet migration to the cloud continues, driven by people arguing that doing it yourself is too complicated and expensive. Let’s see how long until one outage takes down the global economy for multiple days or weeks.
"The cloud" isn't the problem nor is migration to it. The problem is single points of failure in a vast, super-connected, global network.
To demonstrate my point, say someone like cloudflare opted to get "off cloud" and run their own datacenters. Half the web would still go down if they had some issue in their datacenters. If anything, the economy of scale and resiliency of a huge cloud network is far beyond what any single operator can ensure for their own service. If this wasn't the case, cloud services couldn't be as profitable as they are.
It isn't the panacea people seem to think it is. One critical service going down, regardless of whether it's cloud hosted or not, has ripple effects in the broader network.
Hilariously, I did not know about any outages today during the workday because we discourage cloud service usage and nobody complained about anything breaking. :)
Cloudflare KV is also having an outage. I wonder who is reliant on who here.
Looks like more than KV is having an issue. Just tried to load dash.cloudflare.com and no bueno.
seriously doubt Google Cloud is relying on Cloudflare KV lol
Google denies the outage. https://x.com/Google/status/1933246051512644069
"clearing cache and cookies"? what is this, 1997?
First, check that nobody else in your family is making a call on the phone line that your modem is connected to, then make sure to disable your Internet Explorer add-ons before trying again.
1 reply →
lol
for those who boycott X:
https://nitter.net/Google/status/1933246051512644069
https://soundcloud.com/ryan-flowers-916961339/the-internet-i...
Was just about to do a demo, but Google Meet was down. Tried to use Jitsi as a fallback, but couldn't log in because Firebase was down too. Ended up using a Slack Huddle, lol.
love how their status page is green with no issues detected!
https://www.canva.com/design/DAGqKquGD-c/xtRObgH1r_4RoulPAys...
For us Cloud SQL instances are toast but App Engine Standard instances are still serving requests. Google Cloud console is borked too, mostly just erroring out.
some core GCP cloud services are down. might be a good time for GCP dependent people to go for a walk, do some stretches, and check back in a couple hours.
Seems like a wider issue at Google than just GCP, the Sheets and Chat APIs are also returning similar "Visibility check was unavailable" errors.
Presumably many Google products run on GCP
Haha, I don't ordinarily spend a lot of time in the Google Cloud Console but just now I was debugging a squirrely OAuth issue with reCAPTCHA failing to refresh several days running. I'm getting this weird page error, and I think, "Is this an issue with my organization? [futz futz futz] Hey wait is GCP actually down?" And it turns out to be the top discussion on HN. XD
Experiencing 504s in Google Meet.
Google Cloud Console won't load.
Can't wait to see how charts are going to look like here on the project we have developed for Maintel https://variable.io/maintel-digital-landscape/. It shows availability across multiple services as a landscape. Expecting to see a lot of spikes tomorrow.
Getting Gateway timeouts on docker hub. Maybe related? I can pull images.
Example: https://hub.docker.com/layers/library/eclipse-mosquitto/late...
Does anyone know if instance-to-instance networking has been affected? My Redis instance has been throwing a lot of connection errors.
We're not seeing any connectivity issues between pods and vms in our vpc, but your mileage may vary.
Thanks
BigQuery is completely dead
Firebase status page has acknowledged it as a "global issue". https://status.firebase.google.com/
A contact in google mentioned to me that some bad update to Google Cloud Storage service has caused some cascading issues affecting multiple GCP services.
The last few times this happened I wouldn't have thought "So this is the day AI takes over".
But this time...
Spotify was not loading, thought my 5G was bad, used YouTube Music instead without issues. Hmmm...
Cloudbuild completely down for us. Getting "Visibility check was unavailable" errors.
Any chance this is the root being that so many different services are effected? https://github.com/kubernetes/kops/issues/17433
https://cloud.google.com/kubernetes-engine/docs/release-note... google did release an update to gcp k8s today, seemingly shortly before the outage
I doubt gcloud would be affected by an aws-specific cni. Unless maybe enough AWS users have a GCP backup environment that they flipped on all at once, but it seems unlikely
good point. I took that as simply the example that they had in front of them but a generic issue.
GCP Artifact registry still down... Not accepting image push and showing 500 status code
Well this explains the issues I've been having with Spotify through the last hour.
Twitch was broken too: https://status.twitch.com/incidents/b79nyp1yhxql
EDIT: Updated link to point to the specific incident.
Is Amazon running Twitch on Google Cloud (at least partially)?
I don't know, at this point I don't know who uses what. This is maybe unrelated but even BunnyCDN has an incident from a few hours ago (https://status.bunny.net/incidents/6g27lbtp67m4).
Seeing how everything seems to be broken everywhere, I'm very much looking forward to the post-mortem.
When Google said GCP is "down", did it affect entire availability zones within a region? For people who designed redundant infrastructure, did your backup AZs/regions keep your systems online?
The outage was global. For my team specifically, a global Identity and Access Management outage meant that our internal service accounts could not refresh their short-lived access tokens and so different parts of our infrastructure began to fail over the course of an hour or so, regardless of what region or zone they were in. Services were up, but they could not access critical GCP services because of auth-related issues which resulted in internal service errors for us.
To give an example, our web servers connect to our GCP CloudSQL database via a Cloud SQL Auth Proxy (there's also a connection pooler in between but that also stayed up). The connection to the proxy was always available, but the proxy wasn't able to renew auth tokens it uses to tunnel to the database, regardless of where the webserver or database happened to be located. To mitigate this in the future we're planning to stop using the auth proxy and connect directly via mutual TLS but now it means we have to manage TLS certificates.
so much for System Design interview and bs gatekeeping...
Surprised no one else mentioned "it's always DNS" yet :-)
"No major incidents" as of 11:37 PDT.
https://status.cloud.google.com/
File that in the status pages worth ~0 category.
Related ongoing thread:
Ask HN: Is Firebase Down? - https://news.ycombinator.com/item?id=44260669
I wonder how many SLAs Google blew out today with this outage.
Damn you Bart Simpson!
https://en.wikipedia.org/wiki/Bart_Gets_Famous
Our GCP workloads are unavailable across several US regions. The GCP console is intermittently unavailable for most pages.
Crossing my fingers for a quick resolution.
My firebase hosting and firestore db are back online, but GCP console and Google SQL instances are still having serious issues as of 7:00pm UTC.
Ahhh, explains why some of my apps are going crazy... Couldn't read a message from my kids pre-school
Thankfully we use AWS at work for everything critical
It looks like more than GCP: outages reported across the board including aws
https://downdetector.com/
About the only thing not down is down detector.
god send omg, imagine down detector is down lmao
anyone know what tech stack they use and where they host
if all services at down at once, no one is thinking or mentioning a potential attack on US cloud providers ? (China or Russia) Maybe ?
GCP status page now reflect the issues, looks like Google Cloud Dataproc, Google Cloud Storage and Identity & Access Management
#HugOps
Having issues with services in cloud run as well
Ah darn it. My Spotify DJ just stopped working.
sentry is down https://status.sentry.io/
xAI having problems, Supabase down, Discord can't upload images to share in chat. Seems like a major backbone outage.
We're investigating right now. Looks like a potential issue with Cloudflare.
You're right, https://www.cloudflarestatus.com/ is showing outage, which cause google gcp outage, and claude outage.
1 reply →
https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
Same here. Even the page to submit support requests is down.
Cloud console does nothing.
They should host their support services on AWS and vice-versa.
I just logged into several of my GCP accts, everything popped up, multiple home regions.. I wonder what % of folks are feeling this right now.
https://downdetector.in/
We're in us-west-1 and seeing issues across Cloud Run, Cloud SQL, Cloud Storage and Compute Engine.
I'm able to login to the GCP dashboard, but it isn't able to find any of my projects.
Yeah their status page is all green nothing to see here (but all production systems are down).
Claude Code is down :( too lazy to do manual conversion from Cocoapods dependency to SwiftPM
Even though BigQuery is not listed in affected services, we see errors connecting to it
It's listed by regions :(
I'm having trouble getting any Street View imagery. Can anyone else confirm?
Yep, street view is not working at all for me
We're experiencing intermittent slowness and timeouts on our GCP everything.
npm registry happen to be hosted on gcp, because that seems to be down as well.
Root cause has been identified and it's being resolved/monitored now
My friends and I are even having trouble getting Rcs text messages to send.
Everything except us-central1 is back up - it's recovering now though
2 hour outage at this point
this aint looking good yall
GPay which is a widely used payment service in India is down as well
India is having a really bad day today
Meet is also down for me right now. Cannot attend any video calls.
Yup, intermittent db connection issues and cloud storage problems.
Where are the AI agents?
Poor agents, finally taking a break
The AI is over employed
reCAPTCHA affected? I couldn't log into my local utilities website due to a reCAPTCHA error. Downdetector agrees, but I interpret that site as dubious.
Yeah recaptcha is down intermittently
Not just GCP, most of Googles services are out of action
I'm on a meet, in cal, editing a dozen docs, in GCP, pushing commits and launching containers; it's not clear yet what exactly is going on but it's certainly intermittent and sparse, at least so far
stop it. you're overloading their system by doing three things at once. let the rest of us have a turn.
And THAT, Smithers, is why we wear hardhats on the job.
Is this the new Y2k?
Seems recovering now
What is this Touchable Grass stuff I keep hearing of?
Not just GCP. AWS and Cloudflare too.
Did someone screw up BGP again?
Source? We didn't see anything wrong with AWS here.
Can't upload discord attachments from mobile.
Google Cloud Storage seems to be down or very slow
> Waiting for downdetector.com to respond...
Guess they used Jules to code their services :)
Storage, CloudRun, Firebase...... All down....
Auth, GCP, Windsurf,Augment Code,Udio, the list is endless.
Facebook, Reddit and Hacker News is still up, but thats about it
Yarn package registry also appears to be down.
npm is, registry.yarnpkg.com is only a CNAME to npm
Maybe cloudflare?
Cloudflare status page reports an issue: https://www.cloudflarestatus.com/
having issues with cloudflare as well
if everything down at the same time - No one is mentioning an attack on us cloud services ? ( China or Russia ) Maybe ?
Text messaging for android is broken as well
Gemini API isn't working for me :/
identitytoolkit.googleapis.com is 503-ing on us, my whole customer success team is locked out from our platform
when its going to be fixed , i am seeing now more and more services getting outage started with IAM ?
YouTube was down for me for some time
Text messaging on Android is broken
mapbox maps seemed to be down for a few minutes about an hour ago. I wonder if it is related.
GKE workloads are also affected.
Shameless plug for https://rollbar.com
Good luck out there!
YouTube is also very flakey.
when its going to be fixed i am seeing now more and more services are down?
I just realized that the reason the status isn't updated is cause they can't access it lol.
Don't host status pages (or their dependencies) on your own infra lol.
Seems obvious.
It should be obvious because both AWS and Azure have done this in the past and shown what a bad idea it is…
How do you know that?
If LLMs are down work grinds to a halt until they return. Just the new era now.
is supabase on GCP ? My Supabase projects are down.
Supabase is on AWS, but this is looking like an upstream Cloudflare issue. https://status.supabase.com/
internal systems at google are currently broken.
kaggle not responding correctly, is it related?
Our GCP is down
What region?
I think multiple regions are down. asia-south, us-east atleast are impacted.
1 reply →
Interesting that all Digital Ocean services are fine...
i think it'll be disaster.
sheesh so many side-affected issues accross all systems, maybe big tech companies like google shouldn't have laid off all those engineers.. https://www.google.com/appsstatus/dashboard/incidents/Eab7zG...
but no tech bros, just keep following your ketamine addled edgelord when he did this with twitter..
"All locations except us-central1 have fully recovered. us-central1 is mostly recovered. We do not have an ETA for full recovery in us-central1."
An hour later and everything is a mess in central-1. They seemed to jump the gun on that one. Doesn't matter if some dinky service like "AutoML Vision" is working, if GCS isn't, then they shouldn't post an optimistic message.
Now my api can not connect to PostreSQL...
sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3
when its going to be fixed i am seeing now more and more services are down?
They've now added this as a major incident - before it just was listed under overview
Seems recovering now
Can't reach my nest thermometer, but their status page says it's fine lol
This is pretty crazy :D How did it affect you?
I almost died
1 reply →
the real concerns in life
Well, good luck to all googlers dealing with this, that's not fun :(
Hola
It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e...
"Firebase Data Connect unavailable due to a known Google Cloud global outage"
While the Google Cloud status page https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it.
Edit to add: within 10 minutes of this post Google updated their status page. More curiously the Firebase page I linked to has been edited to remove mention of Google Cloud in the status and now says "Firebase Data Connect is currently experiencing a service disruption. Please check back for status. ".
IIRC status pages drive customer compensation for downtime. Updating it is basically signing the check for their biggest customers, in most similar companies you need a very senior executive to approve the update
On the other side of this, Firebase probably doesn't have money at stake making the update
It is not the status page that drives customer compensation. It is downtime.
11 replies →
Nah, its just some client side caching / JS stuff. Clicking the big refresh button fixed it for me, 15 minutes before OP noted it.
(n.b. as much as Google in aggregate is evil, they're smart evil. You can't avoid execs approving every outage because checks without some paper trail, and execs don't want to approve every outage, you'd have to rely on too many engineers and sales people, even as ex-employees, to keep it a secret. disclaimer: xoogler)
(EDIT: for posterity, we're discussing a "overall status" thing with a huge refresh button, right above a huge table chockful of orange triangles that indicate "One or more regions affected" - even when the "overall status" was green, the table was still full of orange and visible immediately underneath. My point being, you gotta suppose a wholeeee bunch of stuff to get to the point there was ever info suppressed, much less suppressed intentionally to avoid cutting checks)
Something must be preventing them updating the status page at this point. Of course they could still deem it not enough, but just from my limited tests, docker, buf, etc (it may not be GCP that is down, but it is quite the coincidence). are outright down. I'd wager that this is much more widespread.
I'm actually on a bridge call with Google Cloud, we're a large customer -- I just learned today that their status page is not automated, instead someone actually manually updates it!
17 replies →
CF too: https://www.cloudflarestatus.com/
This extra funny that GCP status page even includes a “last updated” time, which is exactly built to convey possible failure to update in cases like this
No major incident as of “ Last updated time: 12 Jun 2025, 11:48 PDT”
Maybe the outage is preventing them from updating that specific page? Hmm
EDIT: Looks like it has been updated now (6:49 PM UTC)
Anytime there is an outage that affects App Engine, Google can't seem to get their status page updated for an extended period of time.
Almost an hour to update the page...
I hope this is the case, or google is super unreliable for production grade work.
:))))))
I asked testing to see if it was up, and it pointed out that Google shows nothing but Nest is showing an outage right now, lol
https://status.nest.com/posts/dashboard
Maybe their dashboard is hosted on GCP and they are displaying a cached version. :-)
More likely they are unable to update their own status page, but in either case not covering themselves in glory over at GCP right now.
AWS has this all the time. If you need to know if a service is down in a region, check for other engineers talking about it on X.
GCP just updated their status
Services are recovering in some locations it seems - Discord is healing
Status pages are PR. It gets the same PR treatment as anything else
lies, from big tech?
say it's not so!
well this explains so much lol
@dang could you merge this and https://news.ycombinator.com/item?id=44260669?
No notifications for mentions, have to email the mods at the hn@ email address.
Do we know if email is still working? kidding-but-not-really-because-gmail…
I think I was a bit optimistic in the response time from mods. This thread won the popularity contest quite well...
Thanks for letting me know about emailing the mods, refreshingly explicit to send email.
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
Take an early lunch
[flagged]
[flagged]
[flagged]
Solana is up ¯\_(ツ)_/¯
seems recovering
Borg and K8s were fighting for resources, so Gemini decided to take out DNS. Now a sysAdmin has to step in.
* just trying to add a little humour. pretty stressfull outage. grarr!!
The cloud enables you to scale. It allows us to distribute systems across multiple regions and data centers. Seems that this is true for outages as well.
The PHP application I wrote as a student running on a single self-hosted server had a higher uptime than any of the cloud providers or redundant system I have seen so far. If you don’t need the cloud for scalability, do it yourself and save yourself the trouble and money. Most companies would be better off investing into some IT staff instead of giving away their systems in the hands of some proprietary and insanely complex cloud environment. You are becoming dependent on someone you don’t know, have no control over and can’t talk with directly. Also the single point of failure is just shifting: from your system to whatever system is managing the cloud. Guess one advantage is that you can shift the blame to someone else…