Comment by cmiles8
5 days ago
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
5 days ago
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
Same happened with FTX and co. during 2020 crypto boom. A lot of rogue stuff sloshing around, but investors were hyperventilating.
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
Yeah I think the truth is closer to the opposite. Jensen has complained that the warnings are discouraging investors.
https://www.businessinsider.com/nvidia-jensen-huang-ai-doome...
AI corp valuations are high because their revenue is skyrocketing, not because of the rogue AI stuff.
2 replies →
There is no such thing as bad publicity. (Technically not true, but definitely applicable in this concrete case, as planned P/E ratios are crazy)
*rogue, rouge = red in French!
Thanks :-)
Is it any more wild than gamestop? Investors think chaos is fun.
With Gamestop (without reflecting the current AI situation), there was a real fuckup which could be exploited with a high probability, and a few people saw it, and exploited it, and became rich. Whatever happened after January 2021 is the problem, when the short squeeze went already away, but somehow people still invest in a slowly dying company.
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
> Isn't this the inevitable end-state of unfettered capitalism?
Aren't capitalism built on exactly the opposite principles? It is not greed that rewarded, but the creation of societal goods, and power granted not to the greediest, but to the most useful individuals.
1 reply →
See https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
Only one is true.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so.
Not that I think we should be credulous about stories where they go rogue.
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
11 replies →
Per Indiana House Bill No. 246, pi equals 4. Whether AI models do things their creators and users had no means of predicting or controlling is an empirical question, and the answer is "yes, they do."
Law != reality. These agents are going rogue, nobody is telling them to hack external systems.
'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing.
17 replies →
If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output.
The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
2 replies →
What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem?
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent attempted to compromise Artifactory [...]
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
Some of the problems were Kobayashi Maru scenarios by design.
Great, so as long as alignment guardrails always work and we don't accidentally tell them to solve an impossible problem, humanity is safe?
Hacking a website falls a little short of threatening humanity.
4 replies →
I'm not saying it's not a problem. I'm saying the labs should take more of the blame.
1 reply →
Rogue means not saying "I can't do it"
it also imparts agency on the machine as a way to abdicate personal, human responsibility
[dead]
Models are incapable of "going rogue".
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
This is pedantic. When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
3 replies →
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does. And the harness does so only because a human did that deliberately.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
3 replies →
The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
It’s trendy to anthropomorphize models these days.
Kinda weird people would do that after pouring all human knowledge into it...
[dead]
"Models doing rogue things" is just another way of saying "complex system operating out of control limits".
Software engineering has until now dealt with deterministic systems that operate on Boolean logic, so dealing with stochastic processes involving unaccounted-for variables has never before been something we've had to deal with. But over in the real world, people dealing with physical, mechanical, operational, or even social systems have had to deal with the failure modalities of stochastic systems forever.
Maybe when working with LLMs as functional tools, it'd be wise to bring some of the wisdom from operations research, quality engineering, etc. into play.
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
Also for fun a satire website i made a while ago: https://killm.pixlw.com/
>Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
I don’t think that really counts as it killing anyone, it didn’t pull any triggers or drop the bombs
1 reply →
coop :: a pen for holding animals of the aviary persuasion
coupe :: style of dessert with ice cream
coupé :: a car model with only two passenger doors
It's almost like vibe coding and quality secops don't mix.
That is not correct. AI is not capable of "going rogue" - that is all marketing and FUD.
The only failings here are boring security and infosec blunders.
So we're safe as long as we secure everything.... as the AI hacks out of one company that is hard to get in to, and another company that is likely probed pretty often.
Cool.
bingo
[dead]