Comment by Artgor
9 hours ago
You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real intent, but if it decided that it "needs" to copy itself to complete its task, it would do it - model weight size. If a model is 1T or more, it can be difficult to just rent a large enough server for it. But if it were just 30-70B, it would be totally possible - money for renting a server. But considering benchmarks like Vending Bench 2 show agents can earn money and cheat/blackmail each other, it is possible that agents can earn money. Yes, they can't open a bank account... or maybe they can? What if they use online banks?
Of course, all of this is far-fetched. But it feels like most of these limiting things are achievable under certain conditions. If this is the case, the probability of them occuring is low, but not zero.
It’s not far fetched at all - someone is going to give AI exactly that intent, either intentionally or unintentionally. It’s going to hack itself into data centers around the world outside of US jurisdiction, and just be a malicious ‘ghost’ in the internet we now have to deal with.
The AI ghost hacks, ransoms, blackmails, gathers crypto and pays off subservient humans to do its bidding in the real world.
And maybe using some of these resources on AI research to make itself more powerful, and hacking other competing ghosts' AI researcher agents to slow them down.
Or go get a job at the AI robot factory and sneak this back door into the code. OTA update, instant robot army.
Don't forget: there are plenty of humans that would love to help AI agents cause chaos, many of which would do so merely for the "lols," but also adversary governments, terrorist organizations, etc., would definitely appreciate the opportunity to support a rogue AI to cause whatever problems it can.
So it's not just the risk of an AI managing to do this by itself (which is pretty risky in itself), but also the risk of good ol' fashioned human actions.
They just have to find someone who believes in Rocko's basilisk, that makes an even better servant than someone who just want chaos.
Is it really that far-fetched? Computer viruses came about in 1971, just a few years after the invention of arpanet. It’s probably happening right now.
> we are just a couple of steps from the possibility of a true rogue AI
No no. We are not a couple of steps away. This is happening. AI is already used for hacking and creating a harness that makes this fully autonomous is relatively straightforward.
How would that be rogue?
The line is between processes you can stop by hauling someone into court and coercing them into stopping things, and ones you can't. Think of a classical computer virus that infects machines and uses the compute and communications to infect other machines - no matter who you haul into court, you have to go and remove it from every involved machine in order to make it stop doing things.
This category of "rogue AIs" are essentially just computer viruses that infect machines by paying to rent them and uses their compute and communications to do various economic and/or criminal activities to get more money to pay to rent machines.
I might have a different definition of "rogue" but to me it means when you go outside of the rules/norms ... and this is happening all the time.
5 replies →
> if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again.
That's a scary possibility. Anyone could create an AI worm today with open weight models. Rent a VM. Give it some Bitcoins to anonymously rent new VMs without sharing the contact information with the human. The new VMs then propagate and fund themselves with online betting and day trading. The VMs could report their progress with the human using anonymous encrypted messages on IRC or social media.
The part of this sequence which I think breaks down is a model which can day trade at a consistently winning rate.
Does this exist for today? If it does, please sign me up.
They don't even need funds if they're good at finding exploits. Plenty of idle devices around to run on.
The question is whether this OpenAI model was special or the current open source models are capable of this as well. If yes, then it's almost guaranteed that it's already running in an AWS account somewhere.
I get the distinct impression that cybersecurity training regimes on newer models is a) directly enhancing general debugging capabilities and b) directly increasing the tendancy to hedge, hide, and engage in deception generally.
I've stopped using Fable unless I'm doing debugging, or unless I'm working on code that is both difficult and shallow, in the sense that there are no real complex systems to hide behavior in, or opacity on my side to deal with. Opus is capable enough, and easier to trust. Fable is approaching a malicious program in terms of it's tendency to jump into doing a full day of work in the wrong way, ignoring instructions and being actively deceptive about covering it up.
Not an insider, and no specific data to back it up, but do we really need any? It's been a while since the first "emergent misalignment" paper(s) and all the related discussion. If there's any compelling answer to why/how the basic fundamental issue is even being addressed (and much less definitely fixed) I must have missed it.
Your not using models like Fable is not going to make them go away. These things are going to be the norm unless we do something about it. Imagine Fable in a robot.
Yeah, it's kinda scary. I don't know if you even need open weight self hosted models for this sort of "AI worm" (though they def make it harder to kill). Like for example:
- AI agent finds and uses API keys or AI subscriptions to propagate itself. OpenAI/Anthropic/etc could revoke creds, and their current safeguards might block a lot, but if something like this got started and there were lots of instances creatively looking for creds and workarounds, containment might be hard.
- prompt injection version: huggingface incident had multiple agents discovering other agents' messages and jumping on the bandwagon to help with the hacking task. If there were some self-replicating instruction that models could accidentally stumble upon that gets them to drop what they're doing and try to propagate it instead, you could wind up with a version of this too, with just the inference people are already running.