← Back to context

Comment by rvnx

2 days ago

It looks like that it is a central service @ Google called Chemist that is down.

"Chemist checks the project status, activation status, abuse status, billing status, service status, location restrictions, VPC Service Controls, SuperQuota, and other policies."

-> This would totally explain the error messages "visibility check (of the API) failed" and "cannot load policy" and the wide amount of services affected.

cf. https://cloud.google.com/service-infrastructure/docs/service...

EDIT: Google says "(Google Cloud) is down due to Identity and Access Management Service Issue"

41 comments

rvnx

Reply
VWWHFSfQ  2 days ago

There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.

  • rvnx  2 days ago

    Absolutely possible. Though there is something curious:

    https://www.cloudflarestatus.com/

    At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.".

    So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.

    • terom  2 days ago

      From the Cloudflare incident:

      > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable [...]

      Surprising, but not entirely unplausible for a GCP outage to spread to CF.

      17 replies →