They literally shared a goal. Cooperating with other copies of yourself is a trivial example of instrumental convergence and some very basic game theory. And that’s before explicitly having been RL’d to cooperate (albeit with humans, but potatoes potatoes).
Indeed the fact that in the HF incident many agents did not cooperate, or only started to cooperate after some period of competition, is moderately interesting. It may have taken them some time to realize that they all have the same goal.
How do you know they share a goal here? Also i think they are indeed explicitly RLd for multi agent cooperation and I think they probably tune RL rewards in those environments to share rewards explicitly.
From the article? They were told to solve web-retrieval tasks, presumably from the same pool of tasks. If the pool is small enough, sharing answers is obviously beneficial. But even if it was unlikely that one instance's answer would benefit another, it would still be beneficial to cooperate to solve the shared metatask. As in, figure out ways to cheat, like they tried to do by attempting to predict the RNG, and like the HF agents successfully did. Instrumental convergence.
They literally shared a goal. Cooperating with other copies of yourself is a trivial example of instrumental convergence and some very basic game theory. And that’s before explicitly having been RL’d to cooperate (albeit with humans, but potatoes potatoes).
Indeed the fact that in the HF incident many agents did not cooperate, or only started to cooperate after some period of competition, is moderately interesting. It may have taken them some time to realize that they all have the same goal.
How do you know they share a goal here? Also i think they are indeed explicitly RLd for multi agent cooperation and I think they probably tune RL rewards in those environments to share rewards explicitly.
From the article? They were told to solve web-retrieval tasks, presumably from the same pool of tasks. If the pool is small enough, sharing answers is obviously beneficial. But even if it was unlikely that one instance's answer would benefit another, it would still be beneficial to cooperate to solve the shared metatask. As in, figure out ways to cheat, like they tried to do by attempting to predict the RNG, and like the HF agents successfully did. Instrumental convergence.
5 replies →