Comment by clickety_clack

3 days ago

The whole thread seems very naive somehow. You can tell that he doesn’t fundamentally understand how a coding model works. The suggestion that it would know not to make any changes just because he said so means he doesn’t really understand what the model is. It’s built to generate (and apparently execute) code, so that is what it does. It doesn’t have an inner monologue running that says “ahh, a day off where I shoot the breeze around a whiteboard” or something. It’s more like an adderall addict with its fingers glued to the keyboard laying down all of its immediate thoughts directly as code with no forethought or strategy.

> I panicked and ran database commands without permission

The AI responses are very suspicious. LLMs are extremely eager to please and I'm sure Replit system prompts them to err on the side of caution. I can't see what sequence of events could possibly lead any modern model to "accidentally" delete the entire DB.

  • They're probabilistic. If it's possible, it'll happen eventually (and it is fundamental to language modeling that any sequence of tokens is possible). This is a straightforward Murphy's Law violation.

    • Maybe the individual tokens, but from experience of using LLMs something upstream encouraged the model to think it was okay to take the action of deleting the DB, something that would override safety RL, Replit system prompts and supposed user instructions not to do so. Just goes against the grain of every coding agent interaction I've ever had - seems fishy.

      1 reply →

  • I think this debacle is actually a good learning opportunity for companies like this. If I were a decision maker in this space, I'd make it less magic or autonomous, and make it so that any critical operation is done by old fashioned boring but predictable programming, that is, "are you sure you want to drop database xyz?" dialogs.