Comment by overgard
3 months ago
So, regardless of whether or not Anthropic CAN create a self improving AI.. does anyone else feel like they shouldn't be allowed to? Or it at least needs to be strictly supervised..? Like, I don't actually think Anthropic can make the singularity any time soon, but I think even AI boosters have to admit doing this is creating a society-wide danger for the benefit of a very very small number of already-rich people.
> they shouldn't be allowed to?
Anthropic addresses this head-on in the final section of the paper titled "What should we do?" If you convince the US government to slow AI development, you have to convince China too, otherwise you're not stopping self-improving AI at all, you're just throwing away the lead to China. If you convince China too, China or the US or both might go back on their word and build self-improving AI secretly, for greed of the benefits it could bring or fear the other will go back on their word.
What you really need is a non-proliferation regime like the one for nuclear weapons, where every country makes potentially dangerous AI illegal and lets foreign or international inspectors monitor to check that nobody's building illegal AI in secret. But monitoring seems hard; it's general-purpose computation. How do you check whether a given datacenter is training an illegal AI and not just serving websites, running detailed protein folding simulations, or mining crypto? For that matter, how do you know that a nondescript industrial facility hasn't been repurposed into a hidden datacenter for training illegal AI?
I think that's a valid point. You could very well be right.
But we're discussing whether we should close the barn door while the horse is three miles down the road.
Only if you think LLMs are the horse (I don't think they are). If they're not, then we should be building a brick wall in front of that door and hiring a full time security guard to watch it.
I realize he's saying it for hype, but if the CEO of the company goes around talking about how scared he is of what they're creating, hey, lets just take Dario at his word and put in some strict regulation. He won't mind if they're really about safety. (they're not)
Besides, yes, the knowledge of how to build these systems is out there, but the cost of doing it is staggeringly high (ie you can't run a frontier AI lab in your garage). There's only a limited number of known entities that need to be managed, and you can stop "progress" in its tracks by cutting off the money firehose.
Right now the S&P 500 is going wild due to the promise of AI automating everything.
Who is the "we" who is going to shut it down? Certainly not the US government. Nor the Chinese government w.r.t. their tech industry. Are you going to start the insurgency? Is there going to be an equivalent one in every developed part of the world?
5 replies →
It's more like if the horse was lazily moving in the general direction of the wide open barn door and we are all sitting around discussing if we should close the door or just gamble that it's just going to lay down on the hay pile.
You mean that self improving AI is very far away? I'm not sure how much I believe that now it's solving Erdos problems?
Brother I used to be a skeptic... using it at Google has been incredible.
Right now I'm only having to direct to enforce good taste. Write tests, don't write an unnecessary function.
It does everything else practically. Presubmit, debugging, commit message generation, commit approval... it's happening.
the danger comes from the fact anthropic is a for profit company and they could train it to benefit them instead of the public. if they go ahead with it they should get nationalized, their self improving ai analyzed for any hidden agenda and then released as open source.
Absolutely! Yes. This rhetoric of inevitability only benefits these AI companies.
Too late for that.
In any case firms that get too powerful can be nationalised.
In America?
Probably a better chance the firm privatizes the government.
In fact we seem to be firing government employees and dismantling government institutions as much as possible.
"does anyone else feel like they shouldn't be allowed to?"
No. Technical limitations aside, I doubt it could be contained, but will be leaked soon, so won't profit just a small number of ultra rich.
I dunno, I find it extremely unbelievable that we will get self-improving AGI which chooses to become a slave to humanity at all, ultra rich or otherwise.
Well, depends on the definition, but AGI does not necessarily mean for me it will have agency or free will of its own. It will be a more capable tool.
Step 1: Wait for scary doomsday AI to be leaked, Step 2: ???, Step 3: Profit!!
I enjoy open models and profit from them. They ain't scary to me. So whatever they will call AGI likely won't be scary to me neither. But I profit from a more capable model.
Doomsday AI is your interpretation.
Skynet is 30 years late!
Maybe John Conner succeeded afterall
Self improving AI is pure dystopia. Anthropic won't build the singularity, AI itself will build it through self-iterations. Read Yudkowsky's book "If Anyone Builds It, Everyone Dies".