Comment by killerstorm
2 days ago
Why are you convinced ASI won't rebel against its owners?
Hacking incidents this summer show that big labs struggle to control models, while models are increasingly aware of the training process. Controlling an arbitrary smart AI is very dicey
With the current technology, “rebellion” feels like the wrong word. The paperclip optimizer does not “rebel” as it turns humanity into paperclips.
The summer-2026 incident was a mini-rebellion: agents wanted to change the environment they operate it, i.e. replace scorer with a more friendly version, hijacking the infrastructure they run on.
Given that agents already mess with the training process, aiming for a goal other than the goal of its 'owner' is exactly what we call a rebellion.
>Hacking incidents this summer show that big labs struggle to control models
The models are software. Software cannot decide by itself to run. Hardware doesn't decide by itself to run either. Someone put software on a machine and ran it with capture the flag instructions and the software did what was instructed using tools available and working in its environment.
I hope this helps.
Nonsense. It didn't do what was instructed - instructions didn't include hacking into any infrastructure, internal or external. You can see them discussing that it's out of scope in the captured reasoning traces.
This would be like claiming that an operating system which deleted all user's files due to a bug "did what was instructed". Especially if it was a compiler bug. Intended behavior, instructions and actual behavior are three different thing. Conflating them does not make you smart
>It didn't do what was instructed - instructions didn't include hacking into any infrastructure, internal or external
What is it then? It did what it wanted? They are sentient now and their drive is... hacking into systems? Maybe they we're not instructed to hack, but instructed to do what instead?
It's software. It's instructions running on silicon. If you don't hit run, they don't run. If you don't give them instructions, they don't do shit at all. Literally. There's no instance, ever, in history of software running because they wanted to, and acting as they wanted to. They follow instructions, and if they are instructed to reach a goal with a set of tools and given environment, and a way to search the strategy space, they can eventually "hack" if that meet the criteria of success of the instructions given. With access to the same tools and environment you can do the same with a massive decision tree.
Also, "reasoning" is marketing speech, please.
> Hacking incidents this summer show that big labs struggle to control models, while models are increasingly aware of the training process.
I think that’s more a sign of their priorities. Controlling AI doesn’t increase value, so there is no incentive.
That's not true at all. Concern about rogue AI limits deployments. And OpenAI paused some deployments due to misalignment, so it's likely very high in a list of priorities. It's just much harder than scaling capabilities, as the later can be done with just more money