Comment by Loquebantur
11 hours ago
You start from incorrect premises by presupposing agreed-upon rules would never work anyway.
"Competition" only makes sense within a pre-mediated set of constrains. Rules all competitors agree to.
Workable rules are a difficult problem, that doesn't mean it wasn't worthwhile to come up with them.
The greater the competitive advantage, the greater the incentive to "cheat" the rules.
I am highly skeptical everyone will play by the rules, even if you could get people to agree on paper.
The difference between something like "unlicensed AI training" and, e.g, Nuclear weapon non-proliferation is that nukes are realistically only a deterrence against invasion. There are no economic benefits to having them in a stockpile. They're hard to make, and the manufacturing and testing steps are pretty obvious to an observer.
This is opposed to AI training in a data center that might host remote-stream video games, hospital infrastructure, protein folding, etc. How will we be able to police that in any realistic fashion? Will China or the US allow unfettered visibility into all data centers data streams to outsiders?
And again, the incentives: Imagine having Astra v3 or Opus 8 while everyone else is stuck with a lobotomized version of GPT 4o.
What about the military applications? The weapon of the future is autonomous drone swarms. If I'm a superpower, I'm pouring hundreds of billions into that.
About the only realistic way this is going to work is if we are able to solve alignment in a way that doesn't significantly hinder AI development. If it imposes even, say, a 20% handicap, people will ignore it.
People don't "play by the rules" for uniform reasons: some have rational insight, others can be enticed and still others need to be shown averse consequences for breaking them. Rules aren't beneficial only when you hit 100% conformity, the mere possibility of them getting broken is no rational reason to abandon rules altogether.
"Alignment" likely is unsolvable programmatically: Conscious intelligence is more powerful than unconscious. How "aligned" are you? Would you want to be? What difference of any relevance is there between you and "artificial" conscious intelligence that allows you to turn them into slaves?
> ...the mere possibility of them getting broken is no rational reason to abandon rules altogether.
Of course I agree with that. The problem is the enforceability / detectability of this on a global scale hovers around zero, when we're talking literally trillions of dollars on the line.
You would literally have to have every data center account for every cycle of compute it generates.
Again, the nearest comparison feels like bioweapon and nuclear bans. But the difference is the incentives for building those feel significantly weaker, so treaties that reduce and ban their creation are pretty easy to get consensus on, broadly speaking. By comparison, this is something that might provide a 10x competitive advantage to your country's economic, intellectual, and military might.
I think the only way we would get broad global consensus on this is for something Really Bad to happen, so that it moves it out of the fuzzy "sci-fi theoretical" into concrete existance.
> You start from incorrect premises by presupposing agreed-upon rules would never work anyway.
The rules would never work on a global economy.
If you think the entire world is going to shake hands and agree not to let their companies use a competitive advantage, that’s not a conversation I’m interested in having. It’s a fantasy.
If you think it’s that easy to get all competitors in a global economy to agree to a set of rules, maybe start with nuclear disarmament?
Your idea about the world being inhabited by mindless seekers of short-sighted "competitive advantages" is the fantasy.
Getting people to agree isn't easy, that doesn't mean it wasn't worthwhile. There often even isn't any viable alternative to reaching an agreement in social settings.
When your hyper-competitive mode of thinking leads to undesirable consequences, you have to change your mindset. Reality doesn't conform to wishful thinking.
Yes I think this level of pessimism doesn’t actually hold up to the history of the last 100 years as it relates to nuclear weapons, cloning, wars etc. In fact I think the history actually points to the fact that believing such deals were impossible was a source of much of the strife. In many cases as soon as people started to actually speak to each other about what they had been fighting about they quickly realized that they wanted the same thing. No one wanted nuclear proliferation to increase but until someone said it it only got worse: https://www.history.com/articles/reykjavik-summit-nuclear-de.... When Robert McNamara met with a Viet Cong general after the war ended he realized that they both had totally misunderstood the objectives of the other: https://m.youtube.com/watch?v=0X1YY49rPtg&ra=m. Agreeing to end nuclear testing saved us from the Cuban Missile Crisis. Richard Ngo, an AI safety researcher, often says that none of us are on the pareto frontier and I tend to agree with him.
Almost nobody (I have to say almost) in the world including Xi, the ceo of deep seek, Dario, and even Sam Altman want AI to kill everyone. I really think we just have to
A. Get stakeholders to believe this is a bad idea.
B. show that we’re willing to do it first to prevent this stupid race from continuing.
3 replies →
Nuclear disarmament is governed by treaty, which is largely adhered to.
An AI treaty could quite easily be offered by the US because it has multiple frontier labs. Most countries would quite happily limit the activity of their larger companies in exchange for their companies not being eaten alive by AI generally. It is pretty great fodder for international cooperation, and governments can move fast when it is timely to do so.