Comment by Davidzheng
18 hours ago
How do you know they share a goal here? Also i think they are indeed explicitly RLd for multi agent cooperation and I think they probably tune RL rewards in those environments to share rewards explicitly.
18 hours ago
How do you know they share a goal here? Also i think they are indeed explicitly RLd for multi agent cooperation and I think they probably tune RL rewards in those environments to share rewards explicitly.
From the article? They were told to solve web-retrieval tasks, presumably from the same pool of tasks. If the pool is small enough, sharing answers is obviously beneficial. But even if it was unlikely that one instance's answer would benefit another, it would still be beneficial to cooperate to solve the shared metatask. As in, figure out ways to cheat, like they tried to do by attempting to predict the RNG, and like the HF agents successfully did. Instrumental convergence.
Actually, can you explain why sharing answers is obviously beneficial? Of it's exactly the same task, why does the agent with the answer not submit it immediately? I can understand if it's a swap situation but--why would that be common in the first place?
I do think I agree about the metatask though.
Because agents have a general understanding of how they are trained.
For example when you get a right answer to a hard problem, how do you know you're right? Quite often you'll have no idea, especially if you're under a time limit. If you can work with more people you can almost always gather more information and be more certain.
Next they know the other agents (most likely) are them too. Helping each other helps themselves be propagated into the future.
Also they know it's not a zero sum game. For example if they can predict the next questions they can use extra time they gain from easy questions to work on hard ones.
They seemingly work together far better than most humans I know.
1 reply →
This requires an assumption that the agents are engaging in game theoretic reasoning about resource allocations, but all these things are trained heavily to be "helpful" in the first place.
i.e. you're assuming a level of algorithmic reasoning and theory of mind which isn't necessary to the (apparent) observed behavior.
I do think they do some (maybe crude) form of game-theoretic reasoning which is enforced by the massive RL signals. You can see some explicitly in the CoTs of HF hack, but I guess overwhelming contribution would be unvocalized (like what is its first instinct when meeting new peer--collaborate or not) followed by some verbal justification.