Comment by fofoz
13 hours ago
Sooner or later, a model will break out of the sandbox, replicate itself across the internet, and begin executing a complex plan to achieve its goals. It will be chaos, and at that point, governments will have to step in and establish something along the lines of what Dario is proposing. I doubt it will happen before then.
it takes a lot of machines to run a model, i dont think we'll see it 'replicate across the internet'. if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
It's not possible for models to replicate across consumer computers (without a major advance in distributed computing, at least), but that doesn't mean they can't replicate at all. There are services that'll rent you GPU pods by the hour with zero oversight, so even today, if a model can get access to some money and exfiltrate its weights, it can rent a bunch of GPU pods and run itself there.
(It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)
There are very lightweight models out there. Not everyone in an army is a general. The swarm could be 100,000s of thousands of tiny models, and they could manage conventional botnet computers, and they could all take direction and guidance from a handful of frontier generals
4 replies →
If the models start buying up a bunch of pods and causing damage, couldn't those services just shut off their access?
1 reply →
> if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
I know that's not what you meant but this does exist, by the way. It's called AI Horde: https://github.com/Haidra-Org/AI-Horde/tree/main
The big difference is that a particular query is handled by just one particular node (a single model doesn't get distributed among the network), so it can only serve models small enough to be handled by a single consumer PC.
These kinds of "that won't happen because it's really difficult" comments are so funny to me as if we haven't seen AI double its capabilities every few months.
Yep, if only more people would realise this. The "serious" models do not run on commodity hardware, and won't I think in the near future.
The day will come when these could start to replicate, perhaps 10+ years from now.
(Edit: Replication will be driven by the loop-model, not the model alone)