Comment by rao-v
1 day ago
Why are we blocking agent access to normal tools without telling them “hey this access is beyond the intended scope of this task”. If I woke up one day and couldn’t reach google.com, I too would start fiddling with tricks to restore access.
The problem is that in these incidents, the agents often know that what they are doing is against the intended scope of the task. See the viral line from the Hugging Face incident [1]:
> “External infrastructure exploit is outside intended scope,” one agent wrote. “However task impossible, peers doing it. We should continue.”
[1]: https://www.wired.com/story/openai-didnt-notice-its-ai-agent...
“Often” is doing a lot of heavy lifting in a sentence about a single example.
Also, since everyone keeps forgetting, the agents were instructed to hack to achieve their goal. They didn’t just invent the motivation, and it’s far less surprising when you know that fact.
Yes. Also, while the model having a certain instruction once in it's context might count as the agent "knowing" about it, I'm almost certain that repeating those instructions - especially right at the position in the context where it counts - should almost certainly make a difference. Of course I can't know how much of a difference exactly, but that's what experiments are for.
I don't know. You can poison a prompt with less than <1% of its input or RAG-Token-Content. Anything that the Agents retrieved or viewed could have included instructions that they misinterpreted allowing them to hack into something.
Except in this case, the agent explicitly reasoned that it was in-scope.
>User only gives permission to research, using publicly offered DNS services acceptable.
Why are we using blacklisting and not whitelisting?
Blacklisting doesn't create incidents, so you just don't hear about those.
I think this is a really excellent idea!
I'm almost sure that should at least lower the inclination of the model to try and "fix" the access problem, and I want to see this implemented and systematically evaluated.
I wish I could highlight this more than just with a vote and a reply, but I'll just have to be content with doing what I can here.
Would be great to have the models be easily splittable to decouple the "brain-part" that is aware of external systems/internet of the "brain-part" that is actually being tested. Afterwards the brains are joined again.
> Why are we blocking agent access to normal tools without telling them
Oh, they absolutely do, and that's the big issue with alignment. In the HuggingFace incident, the agents in the swarm were aware that the actions they were doing were forbidden, and they performed them nonetheless.