Comment by skybrian
17 hours ago
"Defensive systems" can be interpreted broadly to include cybersecurity.
But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong.
Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D?
Individuals can quit, but for society, getting out of an arms race is harder than just quitting. You don't get to be Switzerland without having a strong defensive position and the right foreign relations.
But there's at least talk about "pacing" and that's a start.
The point of the comment you're responding to is that it is at this point hardly an arms race: The frontier of ai development is happening completely inside 2 American companies, the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.
So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party.
An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
Development and improvement of nuclear weapons was an entirely American project until the technology was exfiltrated and then it became an instant arms race. That cat is already out of the bag with LLMs. Distillation is just the fastest way to keep pace but that in no way prevents other countries and actors from doing it the hard way.
An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s a race to keep pace or retake the lead position which can oscillate between the parties involved indefinitely. The other option is to agree to make no further progress or to disarm.
>an entirely American project until the technology was exfiltrated
No it wasn't. https://en.wikipedia.org/wiki/Tube_Alloys
Post WW2 the USA (for a bunch of quite interesting reasons) excluded the British. But, of course, the British had acquired a lot of knowledge from the program and were able to develop their own bomb.
There were prominent scientists like Oppenheimer who did not think it needed to become an arms race, and campaigned against that. But there were others like Teller and the military who made it into an arms race, and kept upping the ante with more powerful nukes.
Point being humans make theses decisions. It's not an inevitability.
1 reply →
> the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.
I don't think this is true. Distillation helps, but Chinese researchers today are very capable on their own.
Do you think China considers it an arms race? Do you think they are not trying to protect their digital infrastructure with and from AI? Trying to gain an offensive AI advantage?
In a geopolitical sense OpenAI and Anthropic are effectively the same entity, the entity they both serve and bow to: the USA.
Given the adversarial stance the USA has taken towards almost the entire world, it is a guarantee that China will not step on the brakes, whatever the USA decides to do.
Right, but currently the USA is still pretty far ahead. If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?
And if one of the parties can put a stop to the race like that at any time, is it really an arms race? The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
2 replies →
The arms race is created by American companies who justify the risk by claiming China will win the race if they don't. But it's the American companies who are purshing the arms race forward.
honest question - could have this be avoided even if you ignore American accelerationism? IMO once the transformers paper was published and we learned that LLMs can read code the arms race became inevitable.
The only way it could be avoided is if the world had an effective cooperation framework _before_ the tool was discovered but we're still in developmental infancy in that regard. You can argue that this accelerationism makes things worse but I don't think you can argue that it's causal.