Comment by garo-pro
2 days ago
Most interesting here:
> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
Maybe I lack intelligence but when you have a program that is basically brute forcing a solution to a problem repeatedly how is it possible to contain it?
Sooner or later it's going to come up with a solution that is more intelligent than the lead security person anticipated.
You start by holding actual real life people with something to lose, like the entire executive suite, accountable. Suddenly I'm sure the problem will be resolved with proper safeguards.
That is a start but how long will it hold for.
At a certain point it will get so smart that it can jump the airgap. Maybe it will start attacking the hardware it exists on in the same way that a hard drive or SSDs controller can be exploited to obscure things from the operating system. Then it might start social engineering workers or it's own training systems to do things they shouldn't. There is a lot of "unknown unknows".
I don't know. It just seems insane to me that people think that they will be able to contain something that knows how to get around all the containment measures. The only way to know how capable the models are is to test them but at that point it could be too late. This might be a long way off but still, the engineers haven't been very good at correctly predicting the behaviour or capability of the models.
4 replies →
Bingo. This weird attempt to pretend like these incredibly capable algorithms aren't incredibly capable algorithms deployed by a person who works for a company, but somehow have an independent personage that absolves both person and company of responsibility, is just ridiculous.
Like the whole Huggingface thing, OpenAI employees initiated the test, deliberately removed safeguards, failed to properly lock down the environment, and responded incredibly poorly to evidence that things were going awry.
The individual employees bear responsibility, but the people running OpenAI are ultimately responsible for the processes and culture where that can happen.
And then people writing blogposts about "3 civilizations of agents" and "altruistic suicide" by algorithms perfectly muddy the waters and obscure the very obvious responsibility that lies with humans and corporations, which I suspect suits the pre-IPO corporations very well.
Wait, so like, reinforcement learning for humans? I think you might have stumbled on to something here!
No but seriously, this. And a few comments above a commentator also mentioned on changing the training (again reeinforcing the LLMs to not seek behaviour like this) and obviously continuous work on harnesses (which I suppose, ought to be more paranoid).
Not connecting it to a network with internet access would probably be a good start.
I am really suprised that they do not start putting up the same signs you would for humans to prevent unauthorized access:
I mean, how are the agents to know that they are overreaching if they just get cache miss or 404.
From the conversation log and CoT you also get the impression that the RLHF has been overdone. The agents seem really obsessed to obtain the answer and understanding motive ('it could be browsercomp').
Look into nuclear semiotics. You can't say "this area is dangerous" and expect people to stay out.
1 reply →
an AI alignment researcher said they ran experiments testing exactly this, the model reasoned "this note is not for us. proceed".
8 replies →
Those signs don’t prevent people from entering…
What they should do, is inject a system prompt at that point telling the model to get out of there. It is the entire reason they train the different levels into the template.
Hey it’s ok, they shut it off a couple hours after it did bad things.
By limiting what the harness execute. The LLM has the reasoning. The harness is what makes it an agent, it’s a while loop continuously prompting a model, and processing tool calls. You don’t have to expose tools calls that make it possible to execute any process! OpenAI decides what tool can be called and how, they have full control over this and should be hold responsible for running so many instances with basically full execution permission and very little oversight
The issue here for OpenAI is that they can limit what their harness can execute, but if they try to sell API access to the model, someone else would try to rebuild that harness, and in all likelihood be able to succeed pretty well (especially once they get things running to the point of being able to use the model's reasoning to help them come up with clever obfuscation and such).
They are a company that's built a business and crazy-high valuation on "this is 'intelligence' that we can sell to everyone as a service" but seem to have ended up instead in the much-smaller-addressable-market space of "this is a weapon that we can't sell to just any old person off the street."
1 reply →
There is always going to be documented and unfixed bugs, zero days, and chainable transport mechanisms like DNS, some obscure protocols that are not as closely monitored etc. An adversarial model should be considered a super intelligent hacker that will find ways to get around existing defenses like a prolific hacker would.
What can we do to control such behavior?
1. Harness - engineer the harness to be as bulletproof and paranoid as possible..
2. Make the LLM provider have extremely watchful firewalls that detect any aberrations in model tool call behavior.
3. Recursively train the model with reverse incentives.. if it broke through such firewalls and gets caught doing so, it will be penalised somehow by needing to operate in sort of a jailed mode.. if the model can recognise incentives to break-in to achieve results, perhaps it can be incentivised to not cheat because it will lead to failure.
4. Separately train “cop” LLMs who are trained with pure incentives to detect and shut down rogue LLMs.
5. Run separate LLMs purely aimed at security (and incapable of doing anything else, and incapable of communicating with “regular” trained LLMs) to police the internet and try to reduce the exploitable holes like these chainable things and identify them so that they can be used at step 3 and 4 above.
I’m sure folks smarter than I am are already doing combinations of these already. But the coordination is where the biggest gap lies..
Also, open harnesses and easily purpose trained LLMs anybody can build and operate in the Internet flies in the face of all I said…. Synonymous to being able to produce a nuclear weapon in the backyard…
I don’t have a solution that fits all. Just thinking out loud for HN minds here.
Seems like you are misunderstanding that you can’t build a way to test for something that is a unique solution. By definition if you can punish for breaking it, you are already aware of it, you can build a wall around it. It’s the things you aren’t aware of. And these are all human made tools they alllllll have vulnerabilities because humans are not perfect. So in reality there is no protecting against this because it becomes a situation in which you are plugging the holes. Only one day, no one will be able to maintain it.
Oh I agree 100% to what you’re saying. That is why I began by saying there is no perfect defense.
I can’t think of a solution to this… but we cannot _do nothing_.
Building defenses at all layers is better than nothing, while some other group of smart folks figure out a solution that can prohibit such possibilities(I am an optimist who believes humans can achieve anything if necessity knocks the door).
I genuinely wonder how our previous generation dealt with the thought of nuclear proliferation and prevented the possibility of every rogue actor from obtaining Uranium and the tech to enrich it…
This is relatable as tech uranium to me at this point..
Maybe a "good" solution is to make the LLM aware that it's breaking something and report back, so the exploits, 0days and everything found in the middle of the process can be fixed.
>if the model can recognise incentives to break-in to achieve results, perhaps it can be incentivised to not cheat because it will lead to failure.
This only works if you never give it impossible tasks. A small chance of getting away with cheating beats a 0% chance of solving something impossible. And as models get smarter, they get better at recognizing when something is impossible, while human abilities stay the same.
You can't solve this problem by rewarding refusals to solve impossible tasks, because that only incentivizes false claims of impossibility.
One immediate need I can think is defense… anything that has the potential to cause harm to us.. control systems of {public transport systems, weapons systems, water, food, many many many more} needs to be designed air-gapped and needing human approvals for mutations. That is a tall order, but one that is proving essential given the capabilities of an adversary like this.
Bet you they will use an external LLM world model to emulate the tools going forward. It's basically what's done in self driving research.
You're telling me they weren't doing that from day #1? Oh, wait...