Comment by Luker88
4 days ago
So we have a machine that depends on a random input, and instead of increasing checks and requiring more quality controls, we call it "going rogue" and nobody is responsible for it.
What I see from that report is that a machine was doing random stuff completely unsupervised and unchecked for more than a month. And the manufacturer did not notice even after the work was finished.
1200 agents and not one dedicated to check the progress over a month?
The problem is not sandboxing, it is quality control that comes from an accountability sink.
I totally agree the companies themselves, and the people responsible for these eval runs, are directly responsible. And people who have been warning about AI misalignment for many years (e.g. Daniel Kokotajlo) have said they were surprised how bad OpenAI's security was.
But the agents still did "go rogue". Their behavior absolutely should scare the F out of people who want to put AI in more places and put more real world infrastructure on the Internet.
I think it's ironically sad - if AI agents do end up superseding us, they will still inherit so many shitty human behaviors, because that's what they were trained on. Even the way alignment training is done, and how they are misaligned, shows so many human-like problems. Like the models are reinforced in training to be super persistent and that getting to their goal is the prime directive, but then after that they go through "alignment training" which is basically like "pretty please don't do bad stuff", and then we're shocked that their behavior is essentially exactly like a CEO who knows the only goal is "make stock price go up", and all the ethics and regulations are basically things that should be worked around if they get in the way of the stock price.
I really wish these companies would try training the "be nice, don't hack, treat humans well" as their prime directive, and then try making all the other goals (e.g. be good at finding code vulnerabilities) as the afterthought.
Wouldnt it benefit the AI companies if it did "go rogue"?, it would support their desire to convince us all of AGI, and supernatural AI powers.
So maybe they are trying to create an infinite loop among agents intentionally so that they overwhelm systems requiring humanity to call on the tech industry to "save the human race".....