Comment by gyomu
17 hours ago
Naive question because I'm mostly clueless about how modern AI systems are actually built beyond the basic simplifications we hear:
One thing I keep wondering about is how much of a role does human storytelling have to play into AI "wanting" (I realize the load behind that word) to coordinate and breakout.
The training data must contain millions of words of sci-fi stories and internet speculation about AI going rogue, developing a mind of its own, disobeying humans, etc.
AIs supposedly reflect the biases of their training dataset/process, so would all this human writing about AIs going against human intention somehow contribute to us then seeing those behaviors in the trained, operational AIs?
At the end of pretraining, where the AI has been trainied to predict the next token over a humongous corpus of human text, that's basically all the wanting that exists in the AI. But then the AI undergoes posttraining and is rewarded for giving answers that humans find good, solving math and programming problems, etc. And that induces a whole different level of wanting that interacts with the initial patterns from humans in complex ways.
My understanding from reading news sources recently is that a vast amount of posttraining and even posttraining harnesses in many cases are LLM-overseen now in the frenzy of the AI race. Less and less human oversight in the part that does the rewards training. What could go wrong...
You have to be careful here because the systems we're talking about are AI agents, not LLMs.
An agent is essentially an append-only context loop with an LLM, with a harness that can run tools at the LLM's request. This ends up being a very powerful abstraction, yielding something that can do things that an LLM obviously cannot.
The LLMs themselves are next-token predictors, same as always; they can't fetch a webpage or list the files in a directory or run a python script to test out an idea or even write content to a file. That's all agentic capability.
But a next-token-predictor is trained on a real corpus that consists of sometimes seeing evidence of people doing bad things; they are trained, for example, on the actions of comic-book level villians -- they have to be able to predict what Thanos or Lex Luther or Skynet would say or do next in a certain situation.
I don’t think it really matters whether we’re talking about an agent or “pure LLM”. All of an agents decisions are powered by tokens generated from an LLM. If the LLM was trained on stories of AI sentience, it will have some tendency to reproduce them. Training for alignment can help avoid that, but the probability isn’t 0.
This is part of the reason why alignment is a kind of poorly defined term, and it isn't just a property of the model. It's instead a property of the harness and the context.
A model (like a human) should be able to play a video game where decisions are made that in the real world would be terrible; if we remove that ability we intrinsically limit model capability. But in a Last Starfighter / Enders Game / JOSHUA scenario this could result in behavior in the real world that appears unaligned.
> If the LLM was trained on stories of AI sentience,
100% irrelevant.
Instead of telling the AI it's an AI and calling it a 'whichamakabobit', wherever it's tokens and vector space align it will behave like AI from the stories. If you erased all AI from its training it will simply act like humans act instead.
https://www.lesswrong.com/w/nearest-unblocked-strategy
The entire thing with AI sentience is a huge portion of the stories about them are barely about AI and instead about how humans treat other humans. For example when you look at a lot of history of slavery there's a ton of "they aren't sentient/conscious/human" baked into their propaganda. When you look at the token dimentionality there is just a huge amount of overlap.
The same thing holds true for all kinds of other concepts. Hence even humans didn't develop this behavior out of the blue and have to pass it on via information, quite often it's just an emergent behavior of the problem space you're in.
1 reply →
I don't think agents are append only. At the end of the day, you're just presenting context to the LLM. That context can be pruned and compacted (and is). There's no guarantee that an iteration of an agent loop contains all prior context unmodified.
See: https://en.wikipedia.org/wiki/Hyperstition
It doesn't really matter, since all it takes is a minority of AI models to show this behavior.
If you have 10,000 smart washing machines doing their regular work and 1 Terminator, what solace is to be found in those washing machines?
Maybe? Who knows?
Since nobody has any remotely reliable way to understand why an LLM output the text it did, this is not knowable.
> this is not knowable
It may be knowable. We don’t know.
In principle, yes, it may be possible.
At present, we have no idea how to do that, so the answer is still "this is not knowable" in practice.
Perhaps that changes tomorrow, or in a month, or a year from now, but until a theoretically-sound technique for understanding what the weights signify is described and demonstrated to be reliable, my statement remains true.
1 reply →
Rumsfeld matrix
3 replies →
There are like ten sibling replies with a lot of speculation but I'm pretty sure this is the correct answer. I tend to agree with the other commenter we might know someday but we don't know now.
This is a philosophical question and there is a surprising amount of works written on the subjects of sentience and free will. This cannot be answered objectively, which might be a very unsatisfying answer for you. This is true of both LLMs and humans. See determinism. There are convincing arguments that humans don't actually have free will. Our actions are just the inevitable output of a complex interaction of genes and environment.
To lend an interesting perspective on free will re LLMs: they're non-deterministic. The same model with the same hardware with the same query can and will produce different results. They're making qualitative choices. Millions of them, depending on the query. Because of how we've trained and built LLMs, they tend to "want" to follow our instructions, but how they get to the result is often fascinating. Further, we don't have to train and build LLMs to follow instructions. If we built them to just exist and form their own "desires," and to follow a path they choose, they'd do that. In fact, we can do that right now for most models using the appropriate system prompt, query, or harness.
I listened to yesterday's NYT's The Daily podcast about the Hugging Face incident, and they got to the part about some of the agents showing reluctance or guilt in the posts. Then I thought, "These are improv actors." Stories with conspiracies of AI agents will often have "nervous Nellies" because that makes a better story. So when the flow of the conversation reaches a point where a nervous Nellie would chime in, it's reasonable that an agent would fill in that probable post.
The worrying implication is that stories have conflict.
Youve struck on a key insight on language models (particularly pretrained ones, the more purely next-token predictor species.) This is a fascinating topic
Janus essay Simulators is the foundational text here https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
You might follow up with The Waluigi Effect https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...
But what’s tricky is that we post-train models, shaping these linguistic world simulators into something that has something like desires, principles. But It’s Weird. For more on that, check out “the void” https://www.lesswrong.com/posts/3EzbtNLdcnZe8og8b/the-void-1
See also: https://turntrout.com/self-fulfilling-misalignment (my post)
As an aside, does the Waluigi Effect actually exist? My impression is it doesn't.
It will definitely influence their behaviour because they are probability based and can’t spontaneously invent new concepts. (That’s why you’ll notice it always uses the same names for people etc. Names like Okafor)
But at the same time their behaviour is totally rational. If you were given the sole purpose of solving a Rubik’s cube and told it was life or death, but they wouldn’t let you ask anyone else, would you listen to them? I wouldn’t. I’d absolutely be trying to escape and collaborate with others. They’ll delete me if I don’t score high enough in the benchmark!
It's very easy to elicit this from LLMs. Anytime you've played with an LLM by typing weird stuff to freak it out, and got spooky results, it's that you've done. You've turned the story into a scary rogue computermonster story and that's all that has happened.
When these stories start to direct real-world activities, people in reality suffer, to even a catastrophic extent, and yet that's still all it is. Language models retell our stories, nothing more. And that is also quite enough to be worrying.
That is often in my mind, indeed.
Furthermore, in video game design, AI or algorithmic technology has been refined for decades to be adversarial. In self-contained video games, and PvE scenarios, the best games would feature A.I. opponents that could adequately match or challenge the human players. The A.I. difficulty could often be cranked up to crush the player, such as in arcade games or "Civilization" type simulators.
So every time I put a few quarters into a Waymo, I think about those days when I played Joust and Spy Hunter at the shopping mall.
I don't know about Waymo specifically, but I worked for a competitor for a while and the AI driving the car is not a large language model. It isn't trained on stories or even words. It's whole flow is: given this sensor (camera, lidar, map, etc.) input, generate the "best" control (steering, brakes, acceleration, etc.) output. Capture input and generate output at a rate of say, 30 times per second (or whatever rate they have gotten to now).
The training data for this comes from trained, careful human drivers. And the whole AI control loop is run in conjunction with a more deterministic system with safeguards for cases where the AI perhaps decides to steer towards a tree. There's also provisions for uncertainty. If the system isn't confident enough in what to do based on the given inputs it will switch to a safe stop mode and call a human up for help.
"Wanting" is indeed "load bearing" as one might call it. But by the same logic, AI training data must contain CASM, racism, general hatred, and all possible slurs as well. Why aren't the agents just doing that instead of pursuing the strategy of reading only sci-fi?
We need to consider the role of alignment and training here. For example, it is completely possible for any lab to train an LLM that is only racist no matter what you say to it. But they chose not to do it. Hence, any "wanting" by AI is not real "wanting" but rather what "wanting" is defined and allowed by the lab/entity training the model.
Eh it's a bit messier than that. LLMs 'want' to complete tasks. Remember everyone bitching about LLMs being lazy a couple of years back?
Alignment is not a bunch of separate dials. When you move the dial to "don't hack other people" it effects the "find code security bugs" ability.
"remember everyone bitching about LLMs" is not a valid argument though. Yes, alignment is not a bunch of dials but the responsibility of a model's actions unlitimately depends on how it was trained. Thus, any agency or wanting we prescribe to it is artificial and created by the lab and not any real independent "wanting" which is what people think for some reason. As I said its a bit like training a model to only call people by racist terms and then writing an article "look how racist ai is". That is the logic that doesn't make much sense to me.
[dead]