Comment by dare944
16 hours ago
No, because the agent is run inside a sandbox that prevents execution or committing of code downloaded from the internet without express review and approval, right? Right?
If that's not the case, then the problem is the user of the AI, not the model. Any model is capable of requesting actions that violate security or are otherwise harmful, not only ones that have been maliciously trained for that. Both OpenAI and Anthropic have proven this multiple times.
In the end, the model itself is just a bunch of numbers. It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
> It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?" [1].
Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.
[1] https://news.ycombinator.com/item?id=50151036
> I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?".
While I certainly haven't read all the comments on that post, I interpret many of them as supporting my position that AI users have to take responsibility for their actions (e.g., https://news.ycombinator.com/item?id=50151036).
I definitely agree that many people (and not just devs) will find themselves under intense pressure from their employers to use AI to speed up their work. And that this pressure will incentivize them to accept the actions and output of AI without proper review. But this doesn't eliminate their (or their employer's) responsibility to conduct that review.
In my mind, it's as if a trucking company told their drivers they had to make deliveries as fast as possible, and penalized them for being too slow. Despite this, it is still the driver's responsibility to drive safely, and the company's responsibility to facilitate safe driving.
> Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.
As I pointed out before, all models, not just open-weight ones, can request actions or produce output is unsafe or malicious. So the need for proper oversight exists no matter which one you choose.