Comment by metalsiliconYT

5 days ago

There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.

Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.

  • That's my view. The end goal is to become Too Big To Fail, the means is stifling the competition and getting military funding.

    • Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.

      35 replies →

    • Its not gonna work.

      At this moment in time, it seems frontier market needs regulation - market for open does not actually.

      Oh.. they dont want regulation on that market? Boohoo!

  • Ban means what? That the entire world (ex USA) can use open-weight models? Not a win.

    • If the US declares LLM models munitions then it makes it much more difficult to easily use them. Illegal use of weapons of war can get you banned from SWIFT.

      Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.

    • Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.

      3 replies →

    • How do you ban even ban them in America with the freedom of speech amendments in the constitution?

  • Cryptography, DeCSS ... we have a long history of trying to block things like this and it simply not working.

    There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.

    I suspect that they are fishing for a case, so a judge can set some precedent.

    • You're going about this backwards. They don't have to ban models. They ban the consolidation of compute without a license. This will be far easier and already has legal precedent.

      2 replies →

  • I agree. I hope developers put their token budgets into open weight models rather than funding the banning of open weight models.

    • I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.

      Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.

  • 100% agree. I think OAI is too, but more subtly.

    Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.

    They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.

    I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).

  • This is wrong. https://www.anthropic.com/news/position-open-weights-models

    • You mean like Google's "Don't be evil" motto? ..

      Why would anyone trust these rent seeking corporations? They are losing millions, they know their funds are at risk, Chinese models are catching up to them at lower costs and investors demand returns.

      It only makes sense for them to go for regulatory capture. They'll probably advocate for severe handicaps on foreign and domestic models including open source (if not an outright ban).

      Once they achieve monopoly, they can do as they please with little competition.

    • They might say that, but it's a bit of a fig leaf next to the implications of everything else they say. They incessantly complain about open weights models, framing them as dangerous and national security threats. The audience for national security rhetoric isn't HN or industry, it's government.

    • lol "We support open weight models!"

      BUT!!!!!!

      "We should not sell powerful chips or chipmaking equipment to China"

      "We should crack down on industrial-scale distillation operations."

      "All sufficiently capable models, open and closed, should go through mandatory safety testing."

> the more evil stuff these agents do the more capable and impressive it makes them look

This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?

  • I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?

    Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.

    • > ... it seems to me they want to be taken over by the government?

      They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."

      At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.

      Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")

    • > I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?

      Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.

      So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.

      These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.

      6 replies →

    • US govt will never buy at the (private aka fake) market price - good luck explaining that away to the public. It'll be political suicide.

      If the US govt takes over, existing employees and shareholders will get screwed.

  • There is no model going rogue. There are models that are prompted "do whatever needs to be done to get X", and given a set of tools and virtually infinite computing power. Then they play surprised when the model breaks out of a flimsy containment (as it was ordered to), with barely any logging, and "do whatever to get X". OpenAI said they intentionally lowered the safety measures for the "ExploitGym" test that lead to HuggingFace attack, and they used the same postmortem to praise their own model capabilities (https://openai.com/index/hugging-face-model-evaluation-secur...), is not that they are concealing it.

    If any of us did a tenth of what they did, we would be already in jail.

    • This is ignoring models "getting out" from all the big players. Alibaba had a model hack infrastructure and mine coin when working on a totally different subject than cybersecurity.

      Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.

      We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?

      3 replies →

  • I think it is intended to provoke regulatory capture.

  • maybe the point is to show that models are capable of x, y, and z and when the time comes you want them on your side.

  • There are two side of the coin. Like with sociopath - they have utility if you think you can control/motivate them.

It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.

This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.

  • Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.

    For example, parents are routinely held liable for their sentient children's actions.

    Corporations are routinely held liable for their sentient employees' mistakes.

    The proliferation of this completely nonsensical theory across HN is disheartening to say the least.

    • the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.

      5 replies →

this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".

that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).

so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.

  • I think one kind of issue with most sci-fi is they explained a world that was, but not near as much how it got there.

    There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?

I remember when all the nuclear energy companies were like "our reactors are so powerful, they might melt down and destroy the entire county, maybe even half the state!" and investors loved it and went wild, and then Chernobyl blew up and everyone thought that the Soviets faked it to make their nuclear power plants look more powerful than western ones.

Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.

I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.

Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(

This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.

We're also at a time with authoritarianism is on the rise.

Not hard to see how this can impact those workers, potentially ruining their lives.

To quote Scott Alexander: If Safety Concerns Are A Plot, It’s A Really Dumb One

https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...

Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.

Correct and people are not weighing heavily enough this point

For technology like this the entire selling point of it is how powerful and capable it is

Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to

I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising

You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”

there's also lazy incentives: the less effort they put into it, the more their ego says how great the agents are.

These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.

It's just a weird techno cult at this point with a mostly functional product.

>There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look.

Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.