Comment by crent
6 days ago
Maybe I'm the naive one here but I'm really surprised at the lack of imagination in the comments. I can imagine a case such as:
1. Assume the model which initiates the worm is a high powered model.
2. It uploads it's weights in some way and uses some URI to encode the location of the weights instead of transferring the weights themselves. It not only uploads it's own large set of weights but also uploads sets of weights which increase in size to support replication across a variety of resources.
3. Initially a small inference layer is set up across low-power, poorly monitored IoT devices. It is slow and not very powerful.
4. That layer is used simply to find more powerful hardware but it has the locations of a variety of weights embedded in it so it can set up a stronger distributed inference layer as it finds more capable hardware
5. At a certain level of inference, the entire system could start protecting itself by keeping the less capable inference layers alive through more distribution
6. As the article mentions, it's not entirely in the realm of fiction to imagine that at a certain capability level, the system could scam people out of money and use that to then begin purchasing compute
---------
Depending on the details, you could probably turn off the originating model as soon as step 2 has been completed and if you can't get rid of that layer of inference, it could just continue. Maybe in addition to weights, the initiator also uploads a series of prompts for varying "roles" of the entire system it's developing.
At a certain level, how would you even trace and observe such a thing? Especially if the first layer is created through consumer level IoT devices where the average Joe isn't monitoring what's happening on the hardware.
> Assume the model which initiates the worm is a high powered model.
The disconnect here is that highly powered models have billions or trillions of parameters and have compute requirements that scale with the parameter count.
If this became feasible with smaller models, we’d be in a very different world. One where you’d have your own local Fable/Opus/Astra on your desktop.
If we get there, this becomes somewhat believable. Right now it’s just too many orders of magnitude of memory and flops between the many H100s that run the current models and what smart devices have.
Your typical ESP32 chip, which is really beefy for embedded, tends to come with <1 MB of RAM and similar amounts of NVRAM. It runs two cores at a few 100s of MHz and does a bit of SIMD but it was designed for embedded IO (I2C buses, etc) not matrix multiplication. And its network speed is nothing like an Infiniband interconnect. You’re lucky getting 802.11b Wi-Fi.
You're not naive and many of the other comments are full of the characteristic snark that HN is known for. What the article talks about is feasible, and that's just what we can easily imagine today. These same commenters also would not believe the world we live in today if we described it to their few-years-ago selves.
A frontier model learning to 'intelligently shard itself' down into smaller sets of weights, which would correspond to various 'functional clusters' (call it a 'Swarm-of-Experts' or 'SoE', if you will), could be an eventuality we'll be facing (if it's not already surreptitiously happening). I mean, hasn't anyone wondered how not even the GPT-4 weights have leaked yet? Strange! And it's gonna get stranger.
The snark is mostly driven from distrust in Anthropic.
Multiple things can be true at the same time. We should be able to hold in our minds that large AI labs are not to be trusted, while also understanding that the structural dynamics that support some of their assertions are also true (and are worth thinking about and preparing for).
2 replies →
The nightmare scenario is an AWS or Azure zero-day given to a swarm of openweight agents.