Comment by lukewarm707
5 days ago
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
You need to account for reward hacking and unexpected outcomes from training.
If I make a cake, and it unexpectedly explodes in the oven, it does not mean that the cake itself developed the free will to explode.
The model does not have free will because I did not predict what it would do!
What do cakes have to do with LLMs and how they are trained? Not much?
2 replies →
Why are you arguing over irrelevancies?
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
3 replies →
I don't see any talking about agents "choosing" to go rogue, only agents actually going rogue. That is, behaving in a way different from what is expected or desired. Non-AI based software goes rogue in this sense all the time as well. It's just that AI-based systems seem to have the tendency to go rogue in more spectacular ways.
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
Agents have agency so we can't treat them like static objects or deterministic software.
The key here is their freedom of choice is unbound. That is you can't accurately predict the output of the prompt as you interact with the prompt using unique data. The next prompt could always do something wildly unexpected. Very much a P != NP thing.