Comment by teraflop
6 hours ago
> May 19, 22:10 UTC - Our automated monitoring detected API health check failures and paged our on-calls, who started investigating the issue.
> At 22:20 UTC on May 19, Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action.
If the timestamps are accurate, what was causing the errors 10 minutes before the account was suspended?
The simplest explanation is just that one or the other of these timestamps is wrong, which wouldn't be a big deal. But if the timestamps aren't known with certainty, it seems very odd to include them in the writeup as though they are certain, even though they are very obviously inconsistent with each other.
> If the timestamps are accurate, what was causing the errors 10 minutes before the account was suspended?
Assuming the timestamps are accurate, Google probably started terminating resources while the account was not "suspended" and only completed that after all resources were disabled.
Or the account started doing something nefarious (assuming one of their customers as root cause, not railway itself) that started causing real problems and Google shut it down.
The problem with not having the data is that it’s easy to make assumptions.
The absence of any explanation for the suspension does seem intentional. If it were me that's one of the first things I would've asked so that I could make sure it doesn't happen again.
The 22:20 timestamp from the body of the post is wrong. The timeline section (where the 22:10 timestamp came from) is consistent with itself, and also contains:
> May 19, 22:19 UTC - Root cause identified: Google Cloud Platform has suspended Railway's production account.
They couldn't have identified the root cause before it happened.
That 10 minutes is likely very normal. Possibly...
* A Google employee messes up a setting (like one of the previous incidents) triggers something that looks like a suspension is warranted and it takes 10 minutes to flow through the process to suspend.
* A Railway customer does something corrupt, or seemingly corrupt, Google's system starts limiting access and take 10 minutes to decide it should be a suspension.
These are even more likely if there is a person in the loop to approve, who obvious did not dig deep enough to see that they should not have done so.