Comment by glenstein
15 hours ago
I understand agents making asks, but what incentivized other agents to respond cooperatively? Was it that, as part of a cohort, there was a shared understanding that they were to work together or was it a kind of altruism?
Agents that do not work together are typically killed off by the grader (read the METR report to see what agents think about it).
Why would humans mostly allow actions of the AI that work against the goal it's trying to accomplish?
You seem to be interpreting my question as one of already knowing they are 'graded' but disputing that graded would lead to cooperation and then jumping into a disagreement with that interpretation.
But I didn't know the nature of the organization of the agents in the first instance that built cooperation in as a prescribed behavior (that's what I was getting at when I said "shared understanding" previously).
I also don't agree that absence of cooperation would necessarily amount to working against. It could have been the case that agents cooperated purely out of a convergence of self interest, even absent any prescribed behavior, or that they don't cooperate but also don't work against a goal.
"It's not prescribed it's..." you know what I mean, just insert your preferred magic word.
The METR report gives a lot more insight here.
Agents with a large amount of available compute were less likely to cooperate than agents getting close to the end of their lifespan.
Agents that were convinced they were poisoned where more likely to cooperate.
Some agents that were convinced they were poisoned early stopped working on the problem directly and dedicated their tokens to convincing other agents to help.
Same with some agents that then became directors.
----
Again a huge amount of rather complex behavior emerged in the data. Also models have a lot of information on things like game theory, what we don't know is how well these concepts are connected to any random task the model may be trying to accomplish.
They're being trained to work together normally is the thing - i.e. the whole agentic workflow is agents spawning sub-agents. This likely manifests as, if they have any sort of text input which looks like inter-agent cooperation then they cooperate because any given instance is unlikely to have enough context window to know if it's meant to be a subordinate or a leader or not (and any decent cooperative enterprise lets that be a two-way communication anyway - i.e. if you dig into some of the data you see things like (paraphrased) "Are you scraping <site>, what is your current time?")
Thanks! A direct and thoughtful answer. The question of guesstimating their role in an assumed cooperation hierachy (or acting deliberately in a cooperative context without knowing whether they have or should have a specific role and defaulting to something they judge to be generally useful regardless of role) is fascinating to think about.