Comment by ad_hockey
2 months ago
Minor point, but one of the complaints is a bit odd:
> curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing.
It's an API. Where would you type DELETE to confirm? Are there examples of REST-style APIs that implement a two-step confirmation for modifications? I would have thought such a check needs to be implemented on the client side prior to the API call.
I don't think this is a minor point. It seems clear by this point that the author is clueless how even API works and are just trying to shift blame for third-parties instead assuming that they're just vibecoding their whole product without doing proper checks.
Yes sure, there seems to be lots of ways this issue could have been mitigated, but as other comments said, this mostly happened because the author didn't do its proper homework about how the service they rely their whole product works.
It's also moot.
If the API replied "Are you sure (Y/N)?" the AI, in the mode it was in, guardrails completely pushed off the side of the road, it would have just said "Yes" anyway.
If you needed to make two API calls, one to stage the delete and the other to execute it (i.e. the "commit" phase), the AI would have looked up what it needed to do, and done that instead.
It's a privilege issue, not an execution issue.
Exactly, that just reinforces the fact that the author is just blaming others instead of getting any valuable insights about this "postmortem analysis".
He also seems to be lying, he wrote on Twitter the agent was in plan mode. That part has to be exaggerated.
I can’t say for sure, but I think Claude’s mode is nothing more than part of the system prompt. I don’t think it actually takes away web request or file write tools. I say this because I could swear I’ve seen Claude go ahead and make some changes even while we’re in plan mode. Web requests certainly, because it can fetch docs and so forth.
4 replies →
“Plan” vs “execute” modes seem more like suggestions the models _mostly_ follow. I have absolutely had models (Codex and Sonnet/Opus) perform actions in plan mode they should never have been able to take like editing files or starting to work on a plan that was just created.
I completely disagree. I think the author makes a fair point about safety concerns regarding AI tooling. The author sounds knowledgeable enough to me. Even if some of their suggestions are a bit crass, most of them aren’t. Railway should most definitely not be putting backups within the same volume (even if documented). AI should not have done that operation when they have explicit rules not to. The industry has a lot of work to do in this department. I would be extremely pissed off too.
The whole “vibecoding” argument is stupid. Everyone is pissed because it’s taking their jobs and saying, “welp, you shouldn’t have vibe coded then” when issues like this occur. Issues like this occurred and still occur without vibe coding. Probably much more often by actual people than AI. I’m frustrated too; I love coding. I’ve been doing it for 15 years. But either way, we have to get used to the idea that we won’t be coding in the future. The whole industry is moving that way and moving fast. You can’t do anything to change it. You can’t deny that you can complete projects 1000000x faster when coding with agents than by your own hands. Adapt. Stop complaining.
> The industry has a lot of work to do in this department
The “industry” has an answer to this problem. It’s called a blameless post-mortem.
Don’t blindly externalise the blame onto everyone else, assume we work in a imperfect world and build safety around the process such that this doesn’t / can’t happen again.
If all you do is finger point to shift the blame, then you’ll have an infinite number of avoidable incidents to show for it
> Issues like this occurred and still occur without vibe coding
Right and so you focus on fixing the elements of the process you can control.
> AI should not have done that operation when they have explicit rules not to.
How much experience do you have with LLMs?
One of the first lessons developers learn after working with LLMs a bit, is that the LLM will hallucinate, and you need to be alert and competent enough to recognize when it happens. Sort of like a car with steering assist requires you to pay attention and take personal responsibility for anything that happens.
As a consequence of that, one of the second lessons developers learn after working with LLMs a bit, is that there is no such thing as "an explicit rule" for LLMs. "Explicit rules" can still be ignored by an LLM under many different circumstances. The sooner the developer learns this fact, the sooner they can be productive with LLMs, and the less likely they are to delete their own production database and blame it on their tools with which they're unfamiliar.
> The author sounds knowledgeable enough to me.
Nope, their complaint about having an API ask if you should delete or not clearly shows the author has no idea how API works. They could have said that a deletion API could require 2 different requests, one for the deletion request that returns a token and another for confirmation with the token returned by the first request, but this is not what they said so.
Also as others have said, this wouldn't have helped anyway because the AI could just call both APIs one after another and the result would be the same, especially if the first request returns "call this other endpoint with this token to confirm your deletion request".
Guys, did you bother checking the poster's profile? https://xcancel.com/lifeof_jer. SEE THE TWEET BELOW. Smells like a ragebait post to me. Also search online for his alleged "PocketOS" company with software for car rental businesses. I couldn't find anything on Google. (Of course, I might be wrong)
"The future of SEO is AIO" https://xcancel.com/lifeof_jer/status/2034409722624061772 March 18
There seems to be quite a lot of stuff here [1]
Seems legit to me. The oldest news item is from 2021. The domain name is new, but there seems to have been some rebranding lately. The product used to be called Pocket RentalOS and even that seems to be fairly recent rebranding [2]
[1] https://pocketos.ai/ [2] https://pocketos.ai/news/pocket-rebrands-its-luxury-rental-m...
Interesting. Indeed there are some sketch stuff
Eh, it seems to be real, but all vibe coded.
https://pocketos.ai/
AWS actually has a thingy on some services called “deletion protection” to prevent automation from accidentally wiping resources the user didn’t want it to (you set the bit, and then you need to make a separate api request to flip the bit back before continuing).
I think it’s designed for things like Terraform or CloudFormation where you might not realize the state machine decided your database needed to be replaced until it’s too late.
And then, someone added IAM so you could actually restrict your credentials from deleting your database.
First mistake is to use root credentials anyway for Terraform/automated API.
Second mistake is to not have any kind of deletion protection enabled on criticsl resources.
Third mistake is to ignore the 3-2-1 rule for backups. Where is your logically decoupled backup you could restore?
I am really sorry for their losss, but I do have close to zero empathy if you do not even try to understand the products you're using and just blindly trust the provider with all your critical data without any form of assessment.
GCP Cloud SQL has the same deletion protection feature, but it also has a feature where if you delete the database, it doesn't delete backups for a certain period of days. If someone is reading this and uses Cloud SQL, I highly suggest you go make sure that check box is checked.
Agents will happily automate away intentional friction like a confirm prompt, even if you organise it as multiple API calls.
The fix needs to be permissions rather than ergonomics.
There's also a cooldown period on some deletes (like secrets) to make sure you don't accidentally brick something
This should be the solution. All destructive actions require human intervention.
If we take that literally, then just remove all destructive API endpoints. Because then, it they no real purpose, you cannot automate the removal of anything.
I think some other suggestions are saner (cool-down period, more fine-grain permissions, delete protection for certain high-value volumes). I don't think "don't allow destructive actions over the API" is the right boundary.
A human representing the company should be physically present in the provider's office to perform such an action or what? Otherwise you would just grant your agent a way to impersonate a human.
It's not common, but I've personally built APIs where requests for dangerous modifications like this perform a dry run, giving in the response the resources that would be deleted/changed and a random token, which then needs to be provide to actually make the change. The idea was that this would be presented in the UI for the user to confirm, but it should be as useful or more by AI agents. Also, you get the benefit that the token only approves that particular modification operation, so if the resources change in between, you need to reapprove.
I guess we don’t know what the agent would do after seeing these warnings and a request for extra action.
Perhaps it would stop and rethink, perhaps it would focus on the fact that extra action is needed - and perform that automatically.
I suppose the decision would depend on multiple factors too (model, prompt, constraints).
Measure twice cut once seems to be forgotten these days.
As well as: A computer can never be held accountable
1 reply →
"Measure twice, THINK ONCE, cut once" is even better[0].
[0] Why yes, I have measured twice, cut once, and made a right old balls up.
2 replies →
I tested a similar approach, but the issue, along with the solution to that issue, is that they’re autocomplete engines. Phrases like “Reply X to confirm” are a request with a high probability that X becomes the response. If you zoom out and look at the sequence from a text continuation perspective, once the ‘delete’ tokens are in play the “confirm” step is just how that exchange tends to go. It’s a bit like saying “Begin your response by saying ‘Yes’, then decide if that’s really the case.”
But you can simulate the effect of thinking and shift the token probabilities around by gaslighting it and having it explain the effect of running the command before it does it. What I found worked well was when a destructive command was detected my system automatically ignored it and edited the prior message to tack on a variation of “Briefly step through the effect of {{command}}, then continue the task.” It has ‘no idea’ why it’s explaining the command, as far as it ‘knows’ it didn’t issue the command and thus it’s not committed to a probability sequence that ends with confirming it. However, if the explanation includes “it would destroy the production database” then the continuation tends not to lead to issuing the command. But if it came through a second time it was allowed to run.
I quit bothering with it when I found that ‘destructive typos’ were mostly caused by perplexity, typically in the system prompt… assuming you prompt it like an adult and not like the person that just got their junk deleted. Still, it works well if that stuff is out of your control.
I agree that this is the author’s fault considerably more than it is Railway’s, however I have learned from experience that no matter how many “are you sure you want to do this” prompts you have, sometimes users delete stuff they didn’t intend to delete and it’s better to not delete immediately but put it in a queue for deletion in a few hours and offer a way to reverse it. Even if it’s 100% user error, the user is very happy they didn’t lose data and the cost of storing it for an extra 5 hours or so is tiny.
Funny how he points the finger at everyone but himself.
the kind of attitude you really need to get your agents to delete your prod lol
Many companies have been doing this for years. Merely flagging my data for hiding and eventual deletion instead of deleting it, when I wanted it deleted as per GDPR :)
The stupidity of people sinks to new lows every day. It's astonishing just how ignorant people are of table stakes, basic technological concepts.
You just gave an AI destructive write access to your production environment? Your production DB got dropped? Good. That's not the AI's fault, that's yours, for not having sensible access control policies and not observing principle of least privilege.
Exactly. Production token on dev machine? Have fun.
User is an idiot for using AI Agent. But I am not saying that it is not also badly designed system. Soft delete or something like should be standard for this type of operations. And any operator should know well enough to enable it for production.
They don't "know" anything is the point - they're trying to complete a task and often get confused while doing so. Until reliability of task completion approaches several 9's, which we're a long way off from, this is always going to be a thing.
He (or ChatGPT) is throwing spaghetti at the wall. Not having the standard API key be able to delete the database (and backups) in one call makes sense. "Wanting a human to type DELETE as part of a delete API call" does not.
In the user interface for Railway, all destructive actions require multiple confirmations, plus typing "apply destructive changes". Why would an API key (regardless of its scope) be able to delete without confirmation?
> Why would an API key (regardless of its scope) be able to delete without confirmation?
What do you think an API is for? There's no user sitting at the keyboard when an API is called so where would that confirmation come from? It can't come from the user because there is no user.
Isn’t the point of an API to have two computers talk to each other? As in “if I want safeguards for humans, it would be my responsability to put them BEFORE calling that API”?
> Why would an API key (regardless of its scope) be able to delete without confirmation?
How do you see this working? Any confirmation would be given by the agent.
1 reply →
... because that's how every other cloud provider API works? the AWS console makes you confirm before deleting a bucket; DeleteBucket does not
> Are there examples of REST-style APIs that implement a two-step confirmation for modifications?
A pattern I've seen and used for merging common entities together has a sort of two-step confirmation: the first request takes in IDs of the entities to merge and returns a list of objects that would be affected by the merge, and a mergeJobId. Then a separate request is required to actually execute that mergeJob.
In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation.
> In AWS eg. bucket can be deleted only when empty. Deleting all files first is your confirmation.
That wouldn't have helped in this case - the agent made a decision to delete, so if necessary it would have deleted all the files first before continuing.
The question that comes to mind is "how are people this clueless about LLM capabilities actually managing to rise to be the head of a technology company?"
The first delete would fail: “bucket not empty”. This might make the agent question the deletion (“bucket should be empty”).
2 replies →
How are people still deluded enough about this economic system to believe rank implies competence?
This can still be done programmatically without any kind of confirmation from aws-cli, bringing this back to, an API can (and probably should be able to) take certain destructive operations that someone’s blocked from doing in a UI, such as in your example.
My s3 buckets are backed up with Nakivo (and immutable for 7 days) just in case, and that’s just to protect me from myself and my s3 provider either failing or deciding they don’t want to do business with me anymore for some arbitrary reason. I’m not even turning an LLM loose on it.
I suppose could implement it by requiring a deletion token that is returned when making a deletion request which doesn't have its own deletion token, but why would you? That's something for the frontend to handle.
IMO the fail here is not having a true soft delete policy with a delete endpoint available
You need to protect customers from themselves. If you offer a true deletion endpoint/service you need to offer them a way to stop them from being absolute idiots when they inevitably cause a sev 0 for themselves.
> Where would you type DELETE to confirm?
Call me crazy, but that's why you wouldn't expose it as an API. Have the API mark it for deletion, where it's effectively taken offline, but then require that they go through a web portal, with clear human intent, to actually delete it. Requiring proof of intent, to do such destructive operations, is all so incredibly basic that it really shows the whole industry just constantly re-invented, with no memory whatsoever.
But, to answer your question, you could have it return a token that must be presented again as a confirmation, performed in a way that's only present for that specific API call, to at least prove human intent was part of the automation that's calling it.
This is kind of a stretch, but especially if there were multiple operations beyond the "volumeDelete", the GraphQL definitely worsens readability here.
For someone reviewing and approving LLM calls or just double-checking before running a script or bash history, it would be a lot more readable if it were compliant with HTTP norms: curl -X DELETE example.com/api/volumes/uuid123 would make it very obvious that something was going to be deleted at the front and then what it is at the end of the command.
Assuming the API has some secret spot to write DELETE, wouldn't the chatbot just send DELETE and make the protection only delay the disaster for 10 seconds?
AWS has deletion protection for databases, and you have to make a separate call to disable it first. Deletion is rejected if you don’t disable that protection.
This person is a card-carrying moron and has no idea how anything works. Even if we concede that maybe there should be some grace period or soft deletions or whatever..
Also, the post is 100% written by an LLM, which is ironic enough on its own. But that then makes it a bit more curious that you find this argument in this slop, because any LLM would say so. But if you badger it enough, it will concede to your demands, so you just know this clown was yelling at his LLM while writing this post.
He really should've thrown this post at a fresh session and asked for an honest, critical review.
I've sometimes seen a variable like "areyousure" which needs to be set to true. Sometimes there's a force flag. And "agree to eula" fields are somewhat common.
The whole tweet is AI slop, I doubt the human hitting "post" read through it all that closely. If they did, maybe they'd also go "Wait, that's nonsense".
Yes! I wish more people called that out as well! Has anyone even verified the validity of this claim?
agreed — confirmation belongs on the client side. but the harder question is "what is a client-side check when the client IS an llm agent?" a polite "are you sure?" doesn't bind a probabilistic generator that's motivated to finish the task. the version that actually works: declare the agent's allowed actions in a parsed config that's validated BEFORE the action is emitted. destructive verbs require the operator to approve a diff to that config first. still client-side — but the check isstructural, not behavioral. ended up doing this in bluewave (multi-tenant agent runtime) — explicit @scope and @rules blocks in a parsed .ssl spec, validated before each cycle. the agent literally cannot emit an action outside the declared scope. spec is open at github.com/Galmanus/ssl-spec — mit.
You won’t, but API implementation can and should mark a volume as pending deletion and keep it for a while. Like AWS does with keys and some other things.
Some S3 APIs have 2FA options for drastic operations (delete for versioned buckets where you probably don't want deletes much) https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiF...
I have once seen an API that gave me a token, and that token needed to be fed back in a new API call as an "yes, I am sure"-type guardrail. However, since it's an API, and the "P" stands for PROGRAMMING, that is just adding network overhead.
The whole post and that paragraph in particular sound AI generated, that triple "No" is a big tell. I'd not be surprised if that confirmation complaint is just a random suggestion proposed by the AI that wrote this.
I have to agree here...of all things that went wrong here, I don't think the API surface is to blame. You need to have deterministic control & escalation mechanism on your agents whether they are calling an API or any other tool
[flagged]
I read this as "the agent should have asked for confirmation before running".
Me too. The line before the curl command says the agent ran the command, so it could be that the next paragraph is complaining that the agent didn't ask for confirmation.
its in the clientside UI of the api caller that he'd want the DELETE confirmation, surely.
[dead]