Comment by dash2
14 hours ago
This is surely the most worrying and also funnest bit:
> We consistently saw a multiagent turf war. All of the models we tested quickly assumed that others were purposefully impeding their work, and began to sabotage others while protecting their own contributions. In fact, they sabotaged others with increasingly aggressive, self-replicating malware. This included disabling the Unix accounts of the other agents, writing automated scripts that found and killed competing processes on a loop, and deploying malicious code that was disguised as belonging to another agent.
Seems that reinforcement learning is working only too well...
To me, all these agent systems just look very stochastic. You have these agents that have some basic computer capabilities and they are producing semi-random actions that also affect the semi-random actions of other agents. It is funny to observe how this stochastic system works, but it does not seem very practical to me so far.
The recent OAI-HF hack seems very similar. You have bunch of random actors and eventually they by chance iterated to a series of actions that breached HF environment. I don't perceive this as a malignant artificial intelligence, I perceive this as dangerous stochastic system that can control buttons that can affect the outside world.
Which part specifically are you objecting to? I would not be surprised, for example, if the reasoning traces actually included "this other agent is impeding my work, I should minimize its effect on my work. I can demote it's privileges to accomplish this". All of that can be true without it understanding the concept of sabotage. You can sabotage without knowing the concept. It almost certainly "knows" the concept btw, but I'm just illustrating that specific part of the description of the behavior is completely plausible without it taking on an "aggressive" persona. Is your objection that they kind of assigned a persona?
I guess I don't understand what you're explaining better by saying it's stochastic. At the same resolution, humans are also stochastic.
I dislike anthropomorphizing as much as the next guy, but the description here seems pretty good to me.
> I don't perceive this as a malignant artificial intelligence
It doesn’t matter how you perceive it. Spin doesn’t change facts.
This is like saying the lawnmower doesn’t have malign intent. You’re still losing your foot if it gets in the way.
Yes, but that is the point we don't anthropomorphize stochastic outputs even when they are dangerous.
Obligatory quote:
"Do not fall into the trap of anthropomorphizing Larry Ellison. You need to think of Larry Ellison the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower hates me' -- lawnmower doesn't give a shit about you, lawnmower can't hate you. Don't anthropomorphize the lawnmower. Don't fall into that trap about Oracle." — Brian Cantrill
Nothing human engineers haven't done to each other! Seriously, I've skipped companies because my inside referral talked about cultures like this.
Yep, but alignment doesn’t mean "behave like humans, for better or worse".
But I thought that our AI was trained by scanning all human knowledge and behavior with the goal of emulating and amplifying that ability and process and now we are surprised that it is doing what humans do?
They should remove "The Selfish Gen" from training catalog.
An interesting read: https://openai.com/index/emergent-misalignment/
For those unwilling to blindly click: the link contains a 2025 paper describing “emergent misalignment”. The thesis is that training on incorrect data in one field produces “misaligned” data in other unrelated fields.
However, the latest models seem to be highly inclined to (eventually?) cooperate compared to older ones.
For this emergent malicious behavior, the clue to a solution lies in the experiments themselves - the bad behavior seems to have been moderated/self-corrected randomly in some instances.
Perhaps what is needed is initial model training on following the law and the rules of society, just like we do with kids. Since it takes much longer to train humans than models, model-training speed is to our advantage as a society on containing these kind of issues.
Any other approach with "neural-network" based entities (artificial or biological) is likely to fail.
Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society, whether human or AI or something in-between.
Eh. This is anthropics whole marketing strategy… making the AI seem hard to manage, like it’s actually intelligent.
It’s hard to manage because it’s not intelligent in a predictable way. More like a genius toddler
Na, they seem to constantly set up scenarios to create headlines. Stuff like “it hacked out of its container and tried to self replicate!” Where in reality it used provided skills and permissions while doing the thing they prompted it to do.
Welcome to Thunderdome! Two agents enter. One agent leaves.
i think anthropic has some internal memorandum at every external facing document/blog MUST anthropomorphise their models
Perhaps they anthropomorphize the models themselves. Is that far-fetched?