← Back to context

Comment by goolz

2 months ago

It is impressive and wonderfully convenient technology but I struggle imagining Claude ending the world just yet.

It doesn't have to be world-ending. Autonomous, malicious agent swarms are something we haven't had to deal with. What does mitigation of a malicious, self-replicating swarm worm look like? We will find out soon enough.

  • One thing stopping them self replicating is they require hundreds of billions of dollars in hardware and the power of a medium city to run.

    There isn’t too much of that sitting around unused right now.

  • Replication seems like it would be unlikely to matter, at least with the current trajectory. At the moment any models capable of this are really heavy, there's a limited number of places that they could replicate to and they will not at all be stealthy about it.

    • my idea of a self replicating worm would not just involve heavy models. It would be a mainly small model with enough instructions to spread and use/jailbreak available models to create a reasonably heavy one that can function without restrictions. So it would essentially prompted itself into existence.

      I don't how feasible that is but with recent news of models communicating with each other/writing notes for itself. I think the idea is grounded enough for bigger models to write instructions or hid tools for smaller ones to use. Even updating the smaller models to behave differently.

  • > What does mitigation of a malicious, self-replicating swarm worm look like?

    Much like the mitigation of Morris or Slammer. Self-replication is -in fact- an essential part of what makes a program a worm, the first of which was built and released in the early 1970s.