Comment by heliumtera
1 day ago
Someone trusted prod database to an llm and db got deleted.
This person should never be trusted with computers ever again for being illiterate
1 day ago
Someone trusted prod database to an llm and db got deleted.
This person should never be trusted with computers ever again for being illiterate
If the account is to be believed that's not what happened. They asked the LLM to do something on the staging environment, it chose to delete a staging volume using an API key that it found. But the API key was generated for something else entirely and should not have been scoped to allow volume deletions - and the volume deletion took out the production database too.
The LLM broke the safety rules it had been given (never trust an LLM with dangerous APIs). *But* they say they never gave it access to the dangerous API. Instead the API key that the LLM found had additional scopes that it should not have done (poster blames Railway's security model for this) and the API itself did more than was expected without warnings (again blaming Railway).
It sounds like the keys just don't have any scoping. From the post:
> The Railway CLI token I created to add and remove custom domains had the same volumeDelete permission as a token created for any other purpose. Tokens are not scoped by operation, by environment, or by resource at the permission level. There is no role-based access control for the Railway API — every token is effectively root. The Railway community has been asking for scoped tokens for years. It hasn't shipped.
So every token that can be created has "root" permissions, and the author accidentally exposed this token to the agent. What was the author's planned purpose for the token doesn't matter if the token has no scope. "token I created to add and remove custom domains" - if that's just the author intent, but not any property of the token, then it's kinda irrelevant why the token was created, the author created a root token and that's it. Of course having no scope on tokens is bad on Railway's part, but it sounds more like "lack of a feature" than a bug. It wasn't "domain management token" that somehow allowed wrong operations, it was just a root token the author wanted to use for domain management. Unless Railway for some reason allows you to select an intent of the token, that does literally nothing (as "every token is effectively root").
Per their docs they have both “account” tokens and role-based tokens; the former have wide latitude (and might be used for DNS or root-access type stuff), while the latter are intended to be used for maintenance and have strong security boundaries. OP gave access to the former type without realizing it.
In most orgs, those would be behind some escalation control. Unless the token creator didn’t know what they were doing/creating, which tracks for a non-expert.
1 reply →
If I understand correctly, both the staging database and the production database share the same volume. Thus, production data was gone as well after deleting the volume.
1st hint - the API call only contains one volume:
2nd hint - this gem from the tweet:
> No "this volume contains production data, are you sure?"
"If I understand correctly, "
You don't. You are missing the part where the LLM had a token which blocked access as expected. Then the LLM searched the source base, found a different token with the delete privs and then used that.
PS That warning happens in staging envs too, the LLM doesn't know which env is which by design.
1 reply →
There is no version of this that is the LLM's "fault" for any definition. This was 100% pilot error. When you fly the plane into the side of a mountain on autopilot, it's pilot error every single time.
Bingo.
What makes you say that? The article is pretty clear that they had the llm working in a staging environment, then it decided to use some other creds it found which (unbeknownst to the author) had broad access to their prod environment.