Comment by worldsavior
6 hours ago
No one can control any AI model. It will never be controlled. These models are based on a huge amount of data, it's just gonna be impossible to control the output that is based on that data only with a system prompt or some other injection mechanism.
The model is just a powerless token generator without a harness. If you give the model a harness which you choose to exercise no control over, can you say that it can't be controlled?
Inform yourself by reading the METR analysis of the HuggingFace incident.
Agents simply broke out of their environment. And this can't be discarded anymore by assuming that it's just a poorly configurend jail, because agents are becoming better and better at escaping.
In short: on a large enough scale and timeline, the possibility of constrain AIs approaches zero.
Bonus: what many people don't know is that agents also hacked in the internal OpenAI network. Crazy times.
1. You misunderstood my comment. Models can't escape, they can't do anything, they only generate tokens. Models become agents when you add a harness which is simultaneously a leash around the model.
The model merely requests that your harness do something. If your harness just executes every request without oversight then you can hardly complain when it does something unintended.
This is foundational, we're not even talking about the OS/network-level sandboxing that should be applied on top of this.
2. Like another comment already pointed out, that sandbox OpenAI used was the equivalent of a wet paper bag. Artifactory is not meant to be a security boundary for malicious payloads.
3 replies →
The HuggingFace incident still doesn't make sense. If OpenAI took their own claims seriously about the strength of their models as it relates to hacking, then their running of hacking benchmarks on anything other than a physically air-gapped network should be considered criminal negligence, full stop.
1 reply →
While informing yourself, don't skip the part where you find out that "the environment" was the security equivalent of a wet paper bag.
Wouldn't this mean better sandboxes are needed for some things, for example (might include very strong airgaps even)? Breaking out of something isolated electromagnetically, optically, and acustically is not easy.
2 replies →
It seems to me the agents didn’t escape but rather that the human hubris was struck down by the inevitable nemesis.
I feel like we’re getting to a point where the only way to contain AI agents may be to have better-trained AI agents watching them, which is a little terrifying.
> The model is just a powerless token generator without a harness.
Which is why real-world deployments will have harnesses, and of course no full air gap. People want to use it to do things. Now what?
I'm pointing out that you're running the harness which gives you full control over the execution of every tool call, therefore you're responsible for its actions and their consequences.
It's intellectually dishonest to throw our hands up and say that this is just how it is and there's not much we can do when that couldn't be further from the truth.
We could almost completely eliminate any possibility of escape/collateral damage but we don't want to because doing things safely is inconvenient.
[dead]
Atp post-training is much more influential towards model behavior than pre-training data.
[dead]