Comment by the_duke

15 hours ago

An Anthropic safety researcher just recently quit with very cryptic messages , saying "the world is in peril"... [1] (which may mean something, or nothing at all)

Codex quite often refuses to do "unsafe/unethical" things that Anthropic models will happily do without question.

Anthropic just raised 30 bn... OpenAI wants to raise 100bn+.

Thinking any of them will actually be restrained by ethics is foolish.

[1] https://news.ycombinator.com/item?id=46972496

“Cryptic” exit posts are basically noise. If we are going to evaluate vendors, it should be on observable behavior and track record: model capability on your workloads, reliability, security posture, pricing, and support. Any major lab will have employees with strong opinions on the way out. That is not evidence by itself.

  • We recently had an employee leave our team, posting an extensive essay on LinkedIn, "exposing" the company and claiming a whole host of wrong-doing that went somewhat viral. The reality is, she just wasn't very good at her job and was fired after failing to improve following a performance plan by management. We all knew she was slacking and despite liking her on a personal level, knew that she wasn't right for what is a relatively high-functioning team. It was shocking to see some of the outright lies in that post, that effectively stemmed from bitterness at being let go.

    The 'boy (or girl) who cried wolf' isn't just a story. It's a lesson for both the person, and the village who hears them.

    • Same thing happened to us. Me and a C level guy were personally attacked. It feels really bad to see someone you actually tried really hard to help fit in , but just couldn’t despite really wanting the person to succeed, come around and accuse you of things that clearly aren’t true. HR got the to remove the “review” eventually but now there’s a little worry about what the team really thinks, whether they would do the same in some future layoff (we never had any, the person just wasn’t very good).

    • Thankfully it’s been a while but we had a similar situation in a previous job. There’s absolutely no upside to the company or any (ex) team members weighing in unless it’s absolutely egregious, so you’re only going to get one side of the story.

If you read the resignation letter, they would appear to be so cryptic as to not be real warnings at all and perhaps instead the writings of someone exercising their options to go and make poems

  • I think the perils are well known to everyone without an interest in not knowing them:

    Global Warming, Invasion, Impunity, and yes Inequality

  • [flagged]

    • Weak appeal to fiction fallacy.

      Also, trajectory of celestial bodies can be predicted with a somewhat decent level of accuracy. Pretending societal changes can be equally predicted is borderline bad faith.

      2 replies →

    • > Let's ignore the words of a safety researcher from one of the most prominent companies in the industry

      I think "safety research" has a tendency to attract doomers. So when one of them quits while preaching doom, they are behaving par for the course. There's little new information in someone doing something that fits their type.

The letter is here:

https://x.com/MrinankSharma/status/2020881722003583421

A slightly longer quote:

> The world is in peril. And not just from AI, or from bioweapons, gut from a whole series of interconnected crises unfolding at this very moment.

In a footnote he refers to the "poly-crisis."

There are all sorts of things one might decide to do in response, including getting more involved in US politics, working more on climate change, or working on other existential risks.

I think we're fine: https://youtube.com/shorts/3fYiLXVfPa4?si=0y3cgdMHO2L5FgXW

Claude invented something completely nonsensical:

> This is a classic upside-down cup trick! The cup is designed to be flipped — you drink from it by turning it upside down, which makes the sealed end the bottom and the open end the top. Once flipped, it functions just like a normal cup. *The sealed "top" prevents it from spilling while it's in its resting position, but the moment you flip it, you can drink normally from the open end.*

Emphasis mine.

Not to diminish what he said, but it sounds like it didn't have much to do with Anthropic (although it did a little bit) and more to do with burning out and dealing with doomscoll-induced anxiety.

> Codex quite often refuses to do "unsafe/unethical" things that Anthropic models will happily do without question.

I can't really take this very seriously without seeing the list of these ostensible "unethical" things that Anthropic models will allow over other providers.

I'm building a new hardware drum machine that is powered by voltage based on fluctuations in the stock market, and I'm getting a clean triangle wave from the predictive markets.

Bring on the cryptocore.

Good. One thing we definitely don't need any more of is governments and corporations deciding for us what is moral to do and what isn't.

Wasn't that most likely related to the US government using claude for large-scale screening of citizens and their communications?

  • I assumed it's because everyone who works at Anthropic is rich and incredibly neurotic.

    • Paper money and if they are like any other startup, most of that paper wealth is concentrated to the top very few.

    • That's a bad argument, did Anthropic have a liquidity event that made employees "rich"?

>Codex quite often refuses to do "unsafe/unethical" things that Anthropic models will happily do without question.

Thanks for the successful pitch. I am seriously considering them now.

> Codex quite often refuses to do "unsafe/unethical" things that Anthropic models will happily do without question.

That's why I have a functioning brain, to discern between ethical and unethical, among other things.

  • Yes, and most of us won’t break into other people’s houses, yet we really need locks.

    • This isn't a lock

      It's more like a hammer which makes its own independent evaluation of the ethics of every project you seek to use it on, and refuses to work whenever it judges against that – sometimes inscrutably or for obviously poor reasons.

      If I use a hammer to bash in someone else's head, I'm the one going to prison, not the hammer or the hammer manufacturer or the hardware store I bought it from. And that's how it should be.

      4 replies →

  • You are not the one folks are worried about. US Department of War wants unfettered access to AI models, without any restraints / safety mitigations. Do you provide that for all governments? Just one? Where does the line go?

    • > US Department of War wants unfettered access to AI models

      I think the two of you might be using different meanings of the word "safety"

      You're right that it's dangerous for governments to have this new technology. We're all a bit less "safe" now that they can create weapons that are more intelligent.

      The other meaning of "safety" is alignment - meaning, the AI does what you want it to do (subtly different than "does what it's told").

      I don't think that Anthropic or any corporation can keep us safe from governments using AI. I think governments have the resources to create AIs that kill, no matter what Anthropic does with Claude.

      So for me, the real safety issue is alignment. And even if a rogue government (or my own government) decides to kill me, it's in my best interest that the AI be well aligned, so that at least some humans get to live.

    • If you are US company, when the USG tells you to jump, you ask how high. If they tell you to not do business with foreign government you say yes master.

    • > Where does the line go?

      a) Uncensored and simple technology for all humans; that's our birthright and what makes us special and interesting creatures. It's dangerous and requires a vibrant society of ongoing ethical discussion.

      b) No governments at all in the internet age. Nobody has any particular authority to initiate violence.

      That's where the line goes. We're still probably a few centuries away, but all the more reason to hone in our course now.

      3 replies →

That guys blog makes him seem insufferable. All signs point to drama and nothing of particular significance.

Codex warns me to renew API tokens if it ingests them (accidentally?). Opus starts the decompiler as soon as I ask it how this and that works in a closed binary.

  • Does this comment imply that you view "running a decompiler" at the same level of shadiness as stealing your API keys without warning?

    I don't think that's what you're trying to convey.