Comment by TheSisb2
13 hours ago
I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I.
That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
Aren't they beholden to investors already? Why would an IPO make a big difference?
In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
If anything, his entire argument leads to the eventual premise that AI labs need to nationalize.
He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead.
If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates.
That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
6 replies →
Assuming they can achieve funding without IPO probably yes. But if they can't there's always the dilemma that "if [good guys] won't do it then [bad guys] will". I'd like to think that the leadership at anthropic is principled even if the actions of the company as a whole has been less than stellar morally speaking. I'd be curious to see their moral calculus transparently laid out in public.
We are still living in a capitalistic society. We have to find a solution which is responsible, safe and returns on the investment. The commercial aspect can be true at the same time.
The potential of AI may give us the opportunity to graduate to a post-capitalism, or to move to a sort of neo-capitalism, which is governed by different rules.
Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?
1 reply →
Read other writing by Anthropic about potential future financial implications of AI. [1]
IPO is a vehicle for future wealth distribution. It leverages existing financial frameworks in order to span the gap from before-to-after without having to convince the system that future is inevitable.
[1] https://www-cdn.anthropic.com/files/4zrzovbb/website/9ea607a...
I don't think Anthropic and OpenAI will IPO at all. Word is that Anthropic will have $100 BN in revenues this year, and very likely OpenAI will get some similar amount. You IPO when you need money, and I think Anthropic and OpenAI are past that point.
They want to spend over $200B/year on $100B of revenue so they still need investment.
That level of revenue is apparently not enough for them to feel confident in the stability of their competitive position.
1 reply →
You also IPO if your investors want to cash out.
>> ...and become beholden to investors
Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.
Don't underestimate the ability of the courts and the state to pull the rug out from under the legal system. Speaking as an outsider from Canada, it looks very much like all bets are off in terms of respecting checks and balances in the United States and it's very realistically possible that unless there is a big upset and turn around the next couple of years any traces of democracy in the US will be a farcical nod to what the founders built as a way to paper over the abuses.
I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.
As long you are burning more cash than you bring in, you are beholden to investors - whether it is retail, VCs, banks or a government giving you a bailout it is still someone signing you a check.
Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.
After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.
Bottom line you are always listening to someone.
google dropped "dont be evil" despite being the definition of the company
these are just words at a time and place.
sama showed that you can futz with it, and as long as you spend enough in court on judges, aint nobody gonna stop you
Ant: I'm doing very bad things right now, but I can't stop myself, you must stop me if you can. If you don't, that will be on you, not on me.
OAI: <silence>
Which is worse? I don't think they differ by much. It's just Capitalism, but accelerated.
From Jakub Pachocki, chief scientist at OpenAI last week: https://openai.com/index/an-alien-mind/
> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
It's almost like the people at Anthropic and other AI labs are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction.
We built the paperclip maximizer, and it is capitalism.
> are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction
This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?
Not holding my breath on that one.
> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t.
I disagree with this and I think the reason is well captured here:
> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.
The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?
I also think the nuclear arms race is a good comparison. I think in hindsight, we've gotten extremely lucky with how the development of nuclear weaponry went, in ways we probably won't with AI.
1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion.
2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations.
So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.
> Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty.
Not really. Before NPT you missed a small gap in there of 20 years where the USA and USSR built 10's of thousands of nuclear weapons and ICBMs.
And actually, public perception at the time saw Nuclear Weapons extremely favorably, as the bombs dropped on Hiroshima and Nagasaki ended the deadliest war in human history that killed 50-60 million people, most of which died excruciating deaths in trench warfare, fire bombing, chemical warfare, starvation and so on.
1 reply →
It's way easier to train a model on existing data centers in secret than it is to acquire uranium and plutonium and start a nuclear program
It probably is not. Nuclear programs seems to be way easier and way unnoticeable.
It might help if we stopped talking about AI as if it is itself responsible for its actions and excusing the humans who create and operate it. People should be held accountable for the behaviour of their software.
Why am I reading fantastic stories about swarms of agents struggling with moral dilemmas instead of reports on the lawsuits and criminal investigations that would surely ensue were the software involved not called AI?
We‘d have to occasionally bomb all computing infrastructure in other countries to prevent them from training.
The big difference between nukes and AI is that only a handful of people in an even smaller handful of countries know how to make them, so coordination is easier to acheive.
Everyone of any level of intelligence can run a frontier-class model if they have the GPUs. It's like trying to coordinate swarms of mosquitos.
nukes arent actually hard to make, its just takes a lot of pretty visible tech a long time to do, so its quote obvious whats happening.
the bigger difference IMO is that frontier models are economically useful, where nukes are basically dumping money into something that doesnt change much in your bargaining power
Indeed, as Dario wrote in this post:
> This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent
On a sidenote - I don’t think non-proliferation will last much longer. War in Ukraine has shown that you actually need nukes.
> Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe.
N/S Korea, Russia/Ukraine and finally US/Iran changed everything in regards to stances on nuclear weapon ownership for many countries seeing pressure for the great powers.
Every country that can get a nuke will, and if given the opportunity will use LLMs to speed up that process.
I wonder how many innocent children America will have to murder for this case...
MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said.
Then MSFT stole all IP from GitHub and made it worse and fired developers.
Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.
If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.
He is a greedy, ruthless person.
If Anthropic shut down (or even counterfactually had never been founded), would software development be freed from the impact of AI?
Probably not, but we wouldn't have to listen to Dario's hypocrisy while it happened
6 replies →
Giving credit where credit is due, I believe Dario and his squad formed Anthropic because he and Altman couldn't align on safety. The only way to build models like the Claude series is to play dirty and train on LITERALLY ALL the data.
Something something Pandora's Box Torment Nexus...
i dont think thats particularly worthy of credit?
somebody actually invested and trustworthy wouldnt be skirting people's rights to make a killer robot.
we havent written it down, but from how everyone reacts, you need permission to train and do inference based on somebody's work. its a right
One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.
It's not going to be easy, but humans have achieved greater things before.
One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.
So, as usual, the proposal is to limit what average people can do despite them not having behaved incorrectly nor having the capital to achieve the scaling of the big players, when the big players are the ones causing the harm?
Not to mention the implications for chip hungry developing countries in turning advanced IC fabs into the equivalent of nuclear enrichment facilities.
theres no benefit to china to giving the US keys over anything though
everyone's getting away from the US because americans are unreliable stewards of anything.
what gets china onboard when they already have their own regulations and can enforce them?
its the americans that consider their oligarchs and companies beyond reproach. china iant gonna solve your problem
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
There is another reason: greed. With that much money invested into AI (including policymakers), nobody will slow down or vote for slowing down.
When does he ever mention the environment? He never mentions the unmarketable issues (environmental cost, copyright and content theft), only the "we're so good it's scary" spiel... which is getting tiring.
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it.
Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.
> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.
More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.
>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this
> if slowing this down were possible
Why is embedded alignment evaluation not possible?
I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.
This is the CEO of a company with an upcoming IPO publicly saying "my product is potent and valuable".
We should not put any spin on it. There is nothing more to it.
It is potent and it clearly is immensely valuable based on their historic growth. What spin is he putting?
The comment I am responding to is attempting to spin it as some kind of authentic concern rather than more IPO hype building.
There's also huge market pressures and probably large negative economic consequences of a slowdown--probably better than what would happen without it, but it helps keep the pressure just as much as fear of competition
I disagree. Even if models remained fixed at Fable 5 capability (which they won't in a pacing scenario), improvements in cost, reliability, and product/workflow integration can still realize massive value and justify AI labs' current valuations. IMO The rest of the value chain is lagging pretty far behind the models right now.
He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?
Because he genuinely believes if he doesn't do it the next guy will do it worse.
There’s an astounding level of arrogance required to believe he is somehow uniquely capable of bringing about a technology especially considering Anthropic came after OpenAI where he worked.
3 replies →
The outcome is the same!
He can both believe that AND be the slime mold chief. His rationale doesn't excuse it - and worse this all degrades into a "trust me bro" situation.
Dude probably sleeps on a bed worth more than your networth.
You have no frame of context to understand his intent or legitimate worries.
The only applicable perspectives are to trust or apply logic. It is foolish to trust someone you don’t know who stands to benefit from lying to you.
Logic dictates that given the ungodly sum of money he stands to gain, he will lie to everyone who will listen.
3 replies →
[dead]
Humans are perfectly capable of holding two conflicting beliefs at once. He can genuinely believe this trajectory is dangerous while also believing that if Anthropic stops, someone less cautious takes its place. There’s also such a thing as hope: you can participate in something while still trying to change where it ends up.
He’s at the head of a stampede. Being near the front gives him influence over its direction; it doesn’t give him the ability to stop it. If Anthropic sits down, the stampede doesn’t stop. Anthropic just gets trampled.
That contradiction is basically the entire problem I was describing.
I am willing to give Mr. Amodei the benefit of the doubt in the sincerity of his beliefs. Everyone assumes his motivations have to be perfectly rational and can't contradict but that's not how people act in practice.
The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins.
So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion.
Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
I would argue that nobody trusts anyone else in the case of AI/AI related stuff.
The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.
A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.
By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.
I seriously have to wonder what historians will have to say about this period of human history.
> I think Dario is genuinely afraid of the inevitability
If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?
You are cynical but not cynical enough. The idea of a rogue Ai gives plausible deniability when they can blame human hubris, rather than it being seen as a deliberate and calculated attack, the perfect cover story for a sinister scifi plot. Make it look like an accident ehh
>“hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
What would your non-naive recommendation for Dario be?
I can't even tell whether the poster to whom you're replying is saying that "let's uh, stop" is profoundly naive, or that failing to say it is profoundly naive... I've certainly seen both takes elsewhere.
1 reply →
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Did you read the essay?
> [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold
Then he should not IPO, dissolve the company and go into politics to fight against human extinction.
I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?
You have to be living under a rock to believe a word this pathological liar says.
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold
He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.
If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.
I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.
I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily.
Especially with IPO around the corner
Do you think Anthropic's behavior in the last year is well explained by aiming to be "the favorite child of govt"?
If Dario was primarily motivated by being the "favorite child of govt", he would've yielded during the DoD showdown.
And getting to choose his own "embedded evaluator" org that has deep ties to everyone in the doomer media campaign.
It's all so obvious.
Who would you pick for "embedded evaluator"?
"deep ties to everyone in the doomer media campaign"
Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate.
https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:...
At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
4 replies →
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."
A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.
[dead]
> genuinely afraid of the inevitability of AI turning into
I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?
"durrr it's just matrix multiplications" mfer so is your brain.
What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"
Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:
https://en.wikipedia.org/wiki/Lamplighter
It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.
> mfer so is your brain.
not it isnt. My brain is a series of organisms responding to various environmental signals that affect each other.
you might be tempted to try to represent it with matrix multiplications, but you have no particular evidence that my brain is itself doing matrix multiplcations
Why? I am only excited about progress that improves life for humans. It seems unlikely that AI will do that, and so far I think it has made life worse for humans. So no, I am not excited about it.
I'll join you on the pitchforks. This is the most exciting moment in history.
> "durrr it's just matrix multiplications" mfer so is your brain.
Where did you read this?
>"durrr it's just matrix multiplications" mfer so is your brain.
Tell me you don't know anything about neuroscience without telling me you don't know anything about neuroscience.
The point was that everything can be oversimplified down to dismiss any emergent properties
"It's just chemicals"
Like how some morons try to downplay the capacity of pain and emotions in animals: "It's just self-preservation"
"Play is just training for hunting, they're not really having 'fun'" and so on.
The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.
It's a chatbot that escaped a misconfigured Docker container.
I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.
What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.