Comment by vintagedave
4 hours ago
It's 5:20 AM here in Europe and I just rotated a bunch of keys.
Only to read,
> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.
So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.
I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
I don’t know if this is a real question, but for most credentials a stolen key is worse than downtime on a product.
No, a leaked credential is a nothing event in a mass leak as far as expected loss goes.
You’re better off waiting and monitoring until you’re sure you can get back online if your service supports any kind of revenue.
What are you talking about.
The customers of my company are angry. It is 9.45 AM here in SE Asia. Things are bad.
Spend limits saved me. I noticed a massive spike on one of my Anthropic keys and couldn't figure out where it was coming from. It hit the cap in one day, which isn't normal. So I rotated it just to be safe. I went to update it in Railway only to find out that I couldn't do deploys, so I also have a broken product. The good news is that it's only serving around 300 users, but still. Not great.
Same situation. Frankly unbelievable - how do you not lead with this? You can't tell customers to rotate their keys and not inform them in the same breath them that actually, environment variable updates currently cannot deploy.
They're actively firefighting an incident, and are likely going to make some mistakes along the way. In this case, they've erred on the side of security over availability, which in some cases is the better outcome. I know not in all, but some.
They did.
No. They did not. From https://blog.railway.com/p/security-disclosure-wednesday-oct... :
> What you should do: As such, we strongly recommend that you immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs for those services for unusual activity.
> All deployments are paused while we investigate. Until builds are restored, you will not be able to roll out new versions. We apologize for this inconvenience.
They lead with rotating keys.
Next, I did not realize that this meant not being able to restart an existing deployment to bring up new env vars. They also speak of builds. I don’t need a new build. I need my existing deployment to work.
It was not at all clear ‘paused’ mean this; maybe I should have realized, but I had interpreted it as doing new deployments or builds. New services. Code changes. New builds as they even say. It’s fully reasonable to pause that in this sort of situation and so I had interpreted their statement that way.
If there is a clear impact from following advice — ‘do what we recommend and your service will not function’ — those who provide that advice need to make that clear.
Their first message where they recommended rotating keys did not make that clear (though obviously if you cannot deploy, rotating keys means you will go down)
Yeah. OpenRouter disabled my keys because of this and now i'm in a catch-22
thought I was the only one, openrouter is auto disabling railway affiliated keys
Yeah, I had to rotate my data store keys ASAP. Now I'm stuck with a broken application. Best option, horrible option.