← Back to context

Comment by VWWHFSfQ

2 days ago

There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.

37 comments

VWWHFSfQ

Reply
rvnx  2 days ago

Absolutely possible. Though there is something curious:

https://www.cloudflarestatus.com/

At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.".

So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.

  • terom  2 days ago

    From the Cloudflare incident:

    > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable [...]

    Surprising, but not entirely unplausible for a GCP outage to spread to CF.

    • voytec  2 days ago

      > outage of a 3rd party service that is a key dependency.

      Good to know that Cloudflare has services seemingly based on GCP with no redundancy.

      16 replies →

  • whatevertrevor  2 days ago

    Doesn't cloudflare have its own infrastructure, it's wild to me that both these things are down presumably together with this size of a blast radius.

    • derefr  2 days ago

      Cloudflare isn't a cloud in the traditional sense; it's a CDN with extra smarts in the CDN nodes. CF's comparative advantage is in doing clever things with just-big-enough shared-nothing clusters deployed at every edge POP imaginable; not in building f-off huge clusters out in the middle of nowhere that can host half the Internet, including all their own services.

      As such, I wouldn't be overly surprised if all of CF's non-edge compute (including, for example, their control plane) is just tossed onto a "competitor" cloud like GCP. To CF, that infra is neither a revenue center, nor a huge cost center worth OpEx-optimizing through vertical integration.

      5 replies →

    • smoe  2 days ago

      Latest Cloudflare status update basically confirms that there is a dependency to GCP in their systems:

      "Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable"

      3 replies →