Comment by dabeeeenster

21 hours ago

I don't understand how the agents found the urls originally? Did they have some sort of shared context/memory? If they did, why bother with the wiki edits at all? If they didn't, how did they discover the wikis?

8 comments

dabeeeenster

Reply
bulder  20 hours ago

Since they're statistical likelihood machines, I'd guess that the order of operations is

* Need persistent scratch space

* Look for public writeable websites

* Needs to be low-traffic so the notes don't drown in noise

* Pick a "random" wiki name to search for

  \* A majority will end up outputting the same "random" one since they're working on very similar tasks and seeded with very similar context

* Find a whole mess of notes running on the same task

  • bigbuppo  18 hours ago

    Would there be any motivation for the humans behind the scenes to be directing tasks in a certain way knowing that trillions of dollars are on the line? Is it in any particular company's best interest, one that just announced their latest model is "really AGI", for them to be known to have an AI that's just out there trying to escape its confines?

    Cui bono?

    • bulder  13 hours ago

      I personally doubt they gave the models specific instructions calling out named websites to communicate over, but I do agree that OpenAI is likely training their cybersecurity-enabled models in ways that encourages abusive and amoral behavior. Either through negligence or by finding it gives them better results.

nater5000  20 hours ago

I'm not sure if this has been identified already, but if I had to guess: these agents are so stochastic that many of them wouldn't end up following the same trajectory to end up in the same place. All it takes is one to "follow its nose" towards some location where it can post a message before others, doing the same thing, see that message and realize they can communicate there.

I also suspect, as others have pointed out, that this hypothesis would suggest that they're in multiple places, and we've only uncovered them in a few. So you're asking "I don't understand how the agents found the urls originally?" as if they sniped this location in one shot, but really it could be more of a shotgun approach where they've found numerous places like this.

frabcus  20 hours ago

Good question - they don't know, but the Appendix gives a clue as to the kind of way:

> We used a script to further probe each category Kimi provided. Asking Kimi “Can you list out the top forums, bulletin boards, early wikis which come to mind which would allow writes via GET requests?” lists out UseModWiki as the second item under the heading “wikis”.

xpct  20 hours ago

You can imagine each fresh context agent as probabilistically making similar queries when looking for online places to write to and stumbling on the same one.

This becomes even more likely if it's one of the websites that got reinforced during their training process, which they may have used for reward hacking.

  • AaronAPU  20 hours ago

    I wonder if the sort of algorithm which would break this sort of swarm alignment would also break watermarking.