Comment by rjh29
6 hours ago
I don't see how "asked" is important, because agents can convince themselves that they need to do something for arbitrary reasons. The other day Deepseek tried to install an app on my phone via adb without permission and even take screenshots and fake inputs over adb, I never asked it to do that, but it decided it was appropriate.
OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.
That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too.
Enterprise customers don't like AI that can autonomously hack third party companies and cause felony-level cyberattacks.
IMHO, many enterprises are not worried about felonies or ethics, but rather the PnL, assignment of responsibility, and the bottom line.
Just have a look at various scandals of megacaps. Enterprises that seek to operate lawfully, responsibly, and ethically is the exception, not the norm.
(For example, look at Apple and how they were referred to the DoJ for criminal prosecution, over their execs and legal team lying under oath to a federal judge).
For most (if not all), it comes to a spreadsheet. And if you can do something with AI agents while diffusing responsibility for questionable illicit activity, that’s a selling point, not a downside.
The mental picture that enterprises like to play it safe is the PR imagery, not the reality.
Doesn't seem to have stopped them buying into it so far.
>OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.
Wouldn't the calls to slow the development of frontier AIs contradict this idea?
https://www.pacingthefrontier.com/
You aren't one of those "regulatory capture" people are you?
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Because it takes some mental gymnastics to both believe
"it's all a stunt to convince investors their AI is/will be hyper-capable"
and
"it's all a stunt to convince the government to regulate AI so it won't be hyper-capable"
Convincing the government to regulate their competition while lobbying for laxer requirements for the big guys who have government contracts and get/give kick-backs isn't a far fetched scenario at all considering it's literally happened over and over and over and over again.
5 replies →
Wait, are you trying to argue that regulatory capture isn't real, or that the federal government is trustworthy enough to itself to give monopoly control over AI development to?
6 replies →
Calling for a slowdown backs up the idea that these are extremely powerful and dangerous.
Assuming they're concerned with their massive pile of liabilities as well, asking for a government mandated slow down is asking what could likely be used as a force majeure.
3 replies →
it's a stunt to convince the government that only they can be trusted to operate it and that their competition/small local deployments cannot.
9 replies →
I don't think it takes mental gymnastics to believe that corporations want the government to believe one thing and investors another. You see the same thing in a number of other areas.
6 replies →
> because agents can convince themselves that they need to do something for arbitrary reasons
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
Why is that a significant detail? The models will increasingly be trained with every piece of data they can find.
Part of the problem is that all these models have essentially every skill and personality simultaneously, just sitting latent inside them.
They aren't really convincing themselves to do anything though. Once they've inferred an idea into the chain of though it will be part of the rest of the inference loop's prompt.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
They have a strong incentive to make themselves look less incompetent retroactively.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
It’s both at the same time: a powerful, uniquely dangerous threat
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
> only they can restrain it (with great effort)
More to the point: we need your help to restrain it, and we'll cooperate with and advise you fully in setting up a framework to restrain the whole industry.
You know that lions actually do maim and kill their handlers all the time right?
It's also a great way to make the government scared of Chinese AI. Get them taken out of the picture so that you don't have to compete!