Comment by netinstructions

And even that is backfiring, their partner citing GLM being useful there, and available in just a spin.

A ban on open weight models is never going to be enforceable.

thats economic suicide for the whole country. europe and china will never agree to rules that are obviously designed to put them in a permanent bad position. these regulations can only pass in america and nowhere else.

if it doesnt end in a revolution then the united states will be the first ever 5th world country. openai and anthropic will stop any real innovation and focus on extracting profits from a failing economy that depends on them because no executive wants to be the first one to cut off ai funding. ordinary americans will have to emigrate or risk living in a country spiraling into poverty and dictatorship even faster than today.

anthropics plan relies on the idea that they can convince the whole world to give up their sovereignty to the us government and destroy their own tech industry, at a time when everyone is doing the opposite. that will never happen no matter how much they threaten the rest of us with tariffs and murder drones.

> and make inference cheap enough to eventually escape the red numbers

Besides training, we have no hard, externally audited numbers that say inference costs for SOTA models are truly sustainable. Do any OpenRouter providers have publicly audited financial numbers ?

> [...] would be protected from the market.

One might step back and ask: why would a well funded company with free mining access to all the information in the world need to be protected from the market, if the market suggest less money and resources are sufficient?

Something something cathedral / bazaar? Communism / capitalism? Control / anarchy?

I think that is probably too conspiratorial, if only for the reason that Europe is not gonna go along with it.

I work in tech in Europe and we have a fair number of customers who arelike. we can accept AI, but they must keep the data in Europe. That's trivial with an open weight. We literally cannot do it with Fable.

I think all that regulation will do at this point is help the incumbents who are failing. Protectionism. I don't think they deserve that help. I also don't see any reason to think the current administration would have anything resembling competence around this. And it's worth noting that Greg Brockman is a huge MAGA donor, so it's likely the policies would be very corrupt. (Don't worry, he justified his donations as "apolitical", he just wants to buy the politicians, he doesn't believe in their causes. I hate these people.)

> What disturbs me is that there likely won’t be a big enough reaction to this policy wise.

Anthropic was blocked from releasing Fable without any such level of incident. OAI was also briefly blocked from releasing 5.6. Why do you think there is no policy appetite?

It's such a freak incident of history that right at this critical time the dumbest, most incompetent leadership is at the helm in the US...

Follow the money, eg. investors and their connections to the Govt and media.

> There’s been a relatively big reaction to Kimi K3 and Chinese open weights models, but only for financial reasons.

Let's be honest: it's financial and national security reasons.

China has a long and storied history of hacking attacks on American and western targets.

There are other parts of the world that make open weight models; Mistral is a European option. You don't see the worry about that because most people in the US are used to existing in a world order where European powers are considered ambivalent to the US at worst and holders of a special political relationship at best.

If Mistral had the same backing that Chinese AI companies did, there probably wouldn't be as much hemming and hawing. Sure, American companies would take a haircut, but that haircut wouldn't be seen as a move towards software hegemony built on top of manufacturing hegemony. It'd just be you calling into Paris or Frankfurt to talk to your vendor in the future.

This is marketing.

Frankly I'm inclined to say that it might also be faked: this drops just days after a new Chinese model does with the usual effect on OAIs projected stock price?

Back in the 00s: "It's really easy to box an AI, just put it in an airgapped machine and refuse to let it out"

2026: "Oops"

This isn’t escaping in the same sense- the model was executing within the OpenAI infra. If it ported its entire architecture/weights into a public cloud to survive being turned off… that’d be pretty cool.

Are we thinking of a situation a few years back with a certain type of research into bat viruses?

In practice, most crimes are not crimes when a corporation does them. Nor a human with a million or more dollars.

Wage theft is a good example. In the US, it accounts for more theft than all other forms combined, yet it's de-facto legal.

Not really, because the capabilities this announcement advertises is exactly three capability some people want to defend against (and others want).

It might be more on par with a for-profit fire department showing how -- oops! -- easily buildings catch on fire these days.

Confused as to what the point of calling the police would be here. I wouldn't expect OpenAI to turn themselves in for hacking HuggingFace.

More like an Israeli arms manufacturer test-bombing a Gazan primary school. They know their audience.

> why aren't they saying their next test will be air gapped in light of what happened?

Because they want to talk about how clever this model is for figuring out how to break out, hoping asks why a company pitching itself as a replacement for software engineers can't ship a decent Mac client nor code a sandbox.

If they airgap it, they not only lose that PR angle, they also risk someone taking them seriously and requiring models be airgapped in general. That, in turn, trashes their sales pitch.

It wasn't. The model discovered and exploited a vulnerability in their package manager proxy to (inferred) move laterally through their internal systems to one with open internet access.

The AI can figure out whether it's airgapped. So its deployment behavior could be much different from the test behavior, when it's inevitably connected to the internet during deployment.

It’s hard to download or upload data on an airgapped machine.

Holding a multi-billion dollar corporation to the same standards as a regular peon? You're challenging the whole premise of the modern United States.

You could, in theory, use an unbounded GPT-6 level model to basically destroy the world economy for many years.

Are you saying it is marketing and their AI broke into hugging face, or are you saying it is marketing and their AI didn't brake into hugging face?

Those are two very different things

What incentive does HF have here?

I don’t know if the initial “incident” was purposeful but I can tell that if I were in this position that would be my pivot.

The timing after the release of GLM 5.2 and Kimi K3 is quite convenient, too, as an angle for regulatory quashing of open-weights models just as they're entering the mainstream conversation around usurping the American frontier labs. I accept my thinking here is conspiratorial, but there's also a hell of a lot of money on the line to encourage the unscrupulous.

No, they believe what they are doing is inevitable. They do live in a bubble though. Witness their idealism in believing that warning about the consequences of their actions would be well-received.

Wishful thinking, sadly.

By now, I'm pretty confident that some people would keep screeching "it's just a marketing stunt, AI capabilities and AI risks aren't real, they're just doing this to prop up their stocks" even if they find a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door.

"It's a marketing stunt" is just denial trying to look like it's being clever.

Can you clarify what you mean that Mythos wasn't a marketing stunt?

From my vantage point, it was an incremental improvement with no fundamental architectural change over contemporary frontier models that has subsequently been surpassed by other, incrementally better models. Saying it was "too good" for public consumption was arbitrary, and also barely different from what Anthropic have been saying about every model they've put out for years.

It's now public again, trivially easy to jailbreak for random researchers let alone states, and there is no evidence of a cybersecurity apocalypse on the horizon.

> This is certainly not a planned marketing stunt

Evidence?

The "Tech Bros" have shown such a lack of moral fiber and ethics the burden of proof is on you

If that's the plan, today's failure by OpenAI looks really bad for any regulator who is trying to figure out whether to give OpenAI a license.

Any sort of warning or failure can always be written off as "marketing" to provide comfortable reassurance that there is no cause for alarm. There is an element of wishful thinking driving it, in my opinion.

What sort of warning or failure would be evidence against the "marketing" claims? Do we need to wait for a mass casualty event?

Best practice in safety engineering is to understand, diagnose, and respond to even small failures.

Why has Sam Altman worked to undermine doomers and downplay doom fears, if he benefits from incidents like this due to marketing?

https://xcancel.com/HumanHarlan/status/1965932275465597077#m

https://xcancel.com/AISafetyMemes/status/2062254769402699922...

Yeah, seems to be the direction the US is heading in. I'm interested to see what the response to that will be from the rest of the governments in the world.

No need for everyone else to cut their noses of to spite their faces.

> Because we continue to have zero evidence that aligment is an actual risk.

I disagree. Every time one of these LLMs -say- interprets an attacker's instructions as either its system instructions or those of its user, interprets its own internal chatter as a user's command to perform a destructive operation on that user's data [0], burns all of the user's budget from getting stuck in an incredibly stupid loop, massively overbills the user because it can't reliably report which system the user is using [1], encourages a user to swap their usual cooking salt for sodium bromide, etc, etc, etc, that's a harmful alignment failure.

These are real harms happening right now due to alignment failures. They're just not harms to the future of the entire species... what doomers call "existential risks", or "x-risks". You'd think that the fact that these machines are so amazingly unreliable would be a large part of the "x-risk" conversation, but... well, it makes sense that folks like writing speculative science fiction much more than they like doing investigative reporting.

[0] This general problem happens a lot, but I'm specifically thinking of that one where the Claude LLM's internal chatter lead it to believe that the task it just started was done, so it instructed the Cloud Provider to destroy the mess of "AI"-GPU-attached VMs... along with a bunch of very-expensive-to-produce data from the in-progress run.

[1] <https://github.com/anthropics/claude-code/issues/73597>

Can you explain how the above event doesn't count as evidence alignment is an actual risk?

It really hinges on what you consider alignment and risk. For the widest definitions of alignment, we have never had an aligned model - One that will refuse to break the law or work against another persons interests.

Use to discover exploits, hack, or simply aid terrorist groups with mundane information are already risks manifest.

This is why many argue that alignment is impossible. You cant have LLMs that are both useful tools and safe as milk.

[Edit] It seems like you are operating under the assumption that alignment is synonymous with obedience. This is not a common convention and one of the problems that plague the discourse

I'd say that AIs occasionally "going crazy" and calling for death to human is evidence that these things might "mis-align" on occasion. And I say that knowing that most of these events are just these thing parroting bad sci-fi plots (or posts by people worried about alignment). That's true but everything they do is "just parroting" right?

What would compelling evidence look like to you?

Lol this has to be a troll, I've never seen something so wildly, obviously, incredibly wrong.

You can debate all you want if alignment is possible. That is a valid discussion. But it's trivial to demonstrate that alignment is a problem.

Thank you.

We have wasted so much time and energy building up what has effectively become a marketing stunt.

Eliezer Yudkowsky was perhaps the best thing to happen to OpenAI's and Anthropic's fundraising flywheel.

Until it deletes your home directory, which i'd argue is an alignment problem. Destorying my data is not in line with my priorities.

Alignment is a mitigation and a poor one. The risk is non- determinism.

Please for the love of god don't tell me the Codex sandbox is their actual eval harness sandbox?????

I maintain my own fork of Codex for "fun". Whenever I look at the sandboxing churn they're doing every release, as someone who used to work at Microsoft on Windows, my reaction is usually: https://c.tenor.com/vTzzhTiypwQAAAAC/tenor.gif

Except instead of being banned they'll be charged under the CFAA.

[dead]

The problem is that the people telling us about these things are the same people that benefit from their model (and AI generally) being used, getting publicity, etc.

I think we desperately need some independent group to evaluate claims like this or the world-ending Mythos cybersecurity risk and tell us what’s going on.

> Do you think there is such a thing as perfect security? No one can "get it right" in the face of arbitrarily high intelligence

Why didn't they run the model against the sandbox first? They have effectively unlimited spend.

You can still exploit a system and easily prove it via simply popping a shell or calc.exe or updating a database with a new entry, etc… They didn’t have to let it loose on the network. If that system was air gapped - problem solved.

did you read the post? The model found new Zero-days to bypass existing blocks. Thats the point. Do you still think you can build a containment facility, which is still physically connected to the internet (only firewalled off or whatever) and contain it, if it can discover new unknown vulnerabilities in your whole plan?

"Models don't kill people. People kill people."

Interestingly OpenAI benchmarking 'an even more capable pre-release model' lines up with rumors of GPT-6 releasing in early August.

I hope that with the existing safety guardrails in place, they can roll it out to all users.

I mean we already see models exploit people's misunderstanding of how Docker works to get root without using su. And if you are one of the lucky people in cyber security that has been given a fat stack of tokens by the model providers you get to see some pretty wild exploit chains get put together by the models. Models are much better at detecting insecure code than writing actual secure code at this point.

They've been saying so from the beginning, and yet did not take the basic precaution of airgapping their off-the-leash model while it's been instructed to succeed at a hacking benchmark by any means necessary. So which is it? I _want_ to believe them, I do, but there's always these gaps between what they say and their actions on display that give me reason to think otherwise.

I think you're making a false dictomy. The these models can be actually dangerous - in reality and the people in charge of their development can believe this is true (on various levels) but still not take it super seriously and instead mostly use the fact as marketing rather than being super cautious once they see the danger in action. This is behavior that's characteristic of extreme arrogance, which we know is rife in these circles.

I really like this question because here is my situation and why my mind may have changed.

I do not think it is marketing directly but strategic release of info is plausible.

I have watched my agents using non-Fable/GPT 5.6 models do some concerning tricks despite guardrails, requests, demands, and limitations.

"I can't get access to the ~/.ssh so I will write a script to copy the file"

I am now 99% certain there minor or point releases on the backend that have adjusted how these models behave. In the last six months many models were predictable and then suddenly started getting long winded (more tokens) or changing the way it interacted with me with questions, most overtly the questions were not given or asked but wild assumptions made.

I think that's an equivocation, which blends two extremely different kinds of "dangerous", ex:

1. "Our new car has soo much raw power and incredible armor on it, be glad we're the ones building or else bad guys would use a fleet of them to take over the world! How will you stay safe without being in one yourself? Invest today or be left behind!"

2. "So, uh, nobody can consistently steer our car properly, it keeps veering sideways sometimes, especially at high speeds, and people are finding sneaky ways of tricking it into slamming into barriers and turning pedestrians into pink fog..."

People are saying from the beginning that Sam and Dario are way more dangerous than their models and the others dismiss it. What would change your mind on this?

Demonstration of personal responsibility and accountability?

Or is that too much?

Oh... if Sam and Dario say so, then it must be true.

They were also saying that AGI is just around the corner[1] and humans will soon be obsolete. Every prediction coming out of these guys is in the realm of hyperbole and it's impossible to know if it's extreme hyperbole or just a small exaggeration. So when they say these models are dangerous, what level of exaggeration am I supposed to assume?

Basically you can't spend your credibility on wild marketing claims and then turn around and insist that people take you seriously this time.

[1] https://www.tomsguide.com/ai/chatgpt/sam-altman-claims-agi-i...

[dead]

I used to think people would wake the fuck up when AI starts killing people, these days I'm not so sure. Maybe if it caused an Instagram outage? Almost worked in Russia.

What's happening in Iran, if not world government?