Comment by frio
4 hours ago
> and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
From my timezone, this started last night and is ongoing into today. Has this been going for longer than the status page and email indicated? If so, why was notification delayed so long?
I love your product, have an incredible amount of sympathy for you as humans (having been in similar sleep-consuming nightmares) and was willing to forgive the GCP-triggered outage as an understandable teething pain, but sadly this response has really knocked my confidence.
I understand the hard reality of security issues -- "urgently rotate your keys, they're already compromised" is an important message to share -- but "sorry, that will also require your site going down because we've blocked deploys" (presumably as a response to the incident...) is an incredibly frustrating situation to find ourselves in.
I also want to briefly note that communicating, from the outset, "this is a significant incident, your keys are compromised, the attack is ongoing and you cannot enter new keys as _those too will be compromised_" is a better response than leaving people to figure out the unfortunate combo endpoint of "my keys are compromised, I have rotated them, this has broken my site but I have new keys, and oh surprise, I can't deploy". The full story needed to be communicated more clearly; the email just said "revoke your keys", which I urgently did... and then remembered/found out that changing environment variables requires a deploy.
I would rather have known earlier that I should put a notice on my site, or my own social media, or notify my own customers ASAP and disable some functionality (or make my own choice about porting somewhere else as a mitigation) than get caught out like this after the horse has bolted and it turns out you've been desperately trying to get it back into the paddock for days.