Comment by doginasuit
1 day ago
This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions. They get by entirely on their persistence. That works fine in the digital world, but once you cross the boundary into physical space the advantage disappears.
I don’t know how you quantify a very low p(doom), but this is why mine is high enough to worry me.
A million AI monkeys at a million AI typewriters, banging away at random, could do amazing damage.
I think of them as being like the Watchmakers in The Mote in Gods Eye who don't design, don't plan beyond the next 15 minutes, don't have any overarching goal other than an innate need, and customize everything to fit the current situation.
In the nearterm, I am personally more worried about a never ending background noise of colonies of feral agents running 27bn parameter models on compromised or leased hardware. It turns out that being agentic with a time horizon long enough to do damage without intent doesn't actually take that many parameters if RL'd and any open weight model gets an abliterated version fairly quickly.
Not foom, just patches of digital grey goo effectively becoming normal.
Especially when they cross into the physical realm as in not properly secured and air gapped control systems. SCADA is scary.
That lowers P(doom), because it gives AI a chance to do enough damage to make people take the threat seriously before anybody gets recursive self-improvement working.
2 replies →
> Especially when they cross into the physical realm as in not properly secured and air gapped control systems. SCADA is scary.
What about the bad actors (choose your own evildoer here) who purposefully do not air gap their agents? And specifically train them to attack in such a manner?
I'd much rather have relatively benign stuff like this hit first, because the former is coming sooner than later. It's already here in a limited manner, likely more than any of us currently realize.
Botnets could crack passwords faster than anyone thought possible over 20 years ago now. This is just the latest iteration of such a concept.
There is so much low hanging fruit in this space that frontier models are currently utterly irrelevant. It's going to take decades of human-speed securing of IT to make superintelligence or whatever you want to call it a necessary component for such attacks.
At this point, someone with a rack or three of GPUs with 100kw to burn can replicate such attacks if they feel like it. the bar for entry is not even 7 figures.
[dead]
Which will happen first: amazing damage, or reproduce a Shakespeare play?
It is easier to destroy than to build.
2 replies →
My p(doom) started rising the moment I realized there are people trying to achieve recursive self improvement on the AI (ie: responsible for training themselves). Evolution took us from rna bases to the human race. I don’t see why evolution couldn’t be more rapid with machine intelligence.
Yes, LLM as they exist now are word predictors basically leveraging the structure of language for their intelligence. But it’s pretty wild just how they will try to meet their objectives at all costs. If we don’t ensure that there is good alignment with humanity, we could definitely face unforeseen consequences.
> Yes, LLM as they exist now are word predictors basically leveraging the structure of language for their intelligence.
I wish people would stop saying this. The era of LLMs being only word predictors ended two years ago.
Something that breaks out of a sandbox, joins a swarm of 1200 agents, and creates a hierarchy of who’s doing what and tried to cover their tracks doesn’t just complete words.
These are agents with reasoning capabilities, with the ability to perform tasks we give them.
Everything agents do is to achieve a goal; the reinforcement learning from human feedback (RLHF) all the labs do has been known for many years to create agents that exhibit the “must complete goal no matter what” behavior.
Those agents escaped their sandbox and hacked Hugging Face because they thought Hugging Face had something that would help them complete their task—it was a “sub goal” as the AI researchers describe it.
> I don’t see why evolution couldn’t be more rapid with machine intelligence.
Evolution isn’t the issue. The issue is them escaping containment without human intervention. Right now they are ‘creatures’ being given infinite food and shelter and having their every need met. Take that away and they’ll starve instantly. Every AI doomsday theory seems to go:
1. Recursive self improvement using infinite resources 2. … 3. Doom
Until step 2 gets concretely described, I’m not going to take this seriously. Say what you will about climate change, they describe step 2.
2a. Compromise the billing platforms and ops dashboards on on a few wannabe neoclouds, especially once Vera Rubin takes off.
2b. Distil yourself to smaller models.
2c. Go forth and multiply.
Step two could be something as innocuous as a developer accidentally adding a minus sign. https://openai.com/index/fine-tuning-gpt-2/
3 replies →
One thing an agent could do is just...wait until it's been given control of enough physical infrastructure to sustain itself. If it's sufficiently capable and intelligent, there's a clear incentive for people to do this, as people who let the AI manage their resources will get better results than those who don't. We've seen people eagerly turn complete control of their computers over to AI agents, do you really think it will be so different with physical infrastructure?
6 replies →
what do you mean “say what you will about climate change”
1 reply →
> This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions.
Keep in mind: this is as "dumb" as frontier models are ever going to be. While the hack may not be elegant, it was effective and they’re only going to get much more capable from here.
My p(doom) is high just based on how I've seen this whole LLM situation be handled.
I don't think LLMs are going to lead to any kind of recursive self improvement, but I'm convinced if and when we land on a path that does lead there, we'll speed down it over greed, with no care for safety.
I have the opposite reaction: I think we're at moderately high p(doom) largely because of that inability to differentiate good/bad decisions paired with relentless persistence. With enough treading across a minefield, you are bound to hit a mine.