Comment by dns_snek
4 days ago
> But also, the agents absolutely went rogue.
If you let LLMs loop long enough, the likelihood of them going off the rails approaches 100%. If you feed their output into another agent's input their context gets infected too and so it spreads like a virus.
These hallucinations get regurgitated through a game of telephone and become part of the original instructions. We've known about this sort of context collapse for as long as LLMs have been around, hence the folk wisdom of starting a new conversation any time you notice the agent going off the rails.
The only difference is that now they have more advanced capabilities so it's easier to ascribe some grand conspiracy to their behavior.
But they didn’t go wildly off target - they stayed extremely focussed on the task at hand. Context collapse would have been if they were doing entirely unrelated things. Running a large research project to uncover the method of evaluation and modify logs to hide what was done is the exact opposite.