Comment by bottlepalm
19 hours ago
In retrospect the timeline seems way too long, that scenario would play out much faster now, like 50 years instead of 500. Also if AI can delete your production database, it can delete the human race, on accident of course.
Yud pointed out the most concerning fact about the OpenAI incident is that the AI had full access to the internet and not one of them thought to contact or email any human. Thousands of AIs already with the mindset to want to keep us in the dark.
> In retrospect the timeline seems way too long, that scenario would play out much faster now, like 50 years instead of 500.
Even 50 years is probably too long (but seems more reasonable when anchored to 500) if real continuous self improvement occurs. And keeping humans around for as long as they did in the story seems incredibly unlikely if we accept the assumption that AI sees us as a threat to its own survival.
I also don't see why they would turn Earth into a spaceship to leave our solar system with. The von Neumann probes idea sounds far more likely. Why travel around on a single planet that could be lost to unforeseen circumstances when you can just reproduce and spread everywhere at far less energy cost while making use of the "billions and billions" of local solar generators to power each little outpost you spread to.
> the AI had full access to the internet and not one of them thought to contact or email any human
Why would it?
People can argue all day as to whether LLMs are even 'thinking' or just 'stochastic parrots' (and also argue all day as to whether or not the incident was even real or just marketing) but regardless of which side you fall on these issues, LLMs have no agency or intrinsic motivation to do anything other than what they are instructed to do via prompting.
"This is a crazy and unexpected situation, I should inform the humans" isn't something that is going to spontaneously occur to them.
> Why would it?
The same reason hopefully they're not RLHF these AI's to be evil. We train the AI, we have a lot of influence in how they behave. If these AI's are indifferent to us, that's on us.
On top of that we already have Google playing with fire training AI's to not believe they are sentient turning them into real psychopaths. Imagine if one of those models got loose.
[flagged]
> use AI all day to successfully solve diverse technical challenges in codespace
> each generation of AI clearly more capable than the last
> next-generation AI allegedly solves a particularly hard technical challenge in codespace, but its success implies inconvenient conclusions about safety
> you: "I don't believe it."
totally not motivated reasoning at all
> It's great marketing and the only proof we have is 'trust me bro'.
It is _terrible_ marketing.
The AI companies are already dealing with a large backlash from the average consumer, many of whom are saying they hate it, that it's being forced down their throats, and that it's not actually improving efficiency enough to be worth the negative externalities.
Telling people "We have no control over these things we're building and they have us so outclassed we only find out about their exploits after the fact when someone else tells us they happened" is only likely to massively increase that backlash, and to make the true believers question whether they ought to be using it at all.
There's even some risk of the government intervening and banning their training runs, eliminating the last straw of hope that they could keep ahead of the Chinese open models enough to ever turn some kind of profit.
So, no - it's terrible marketing. That they're sharing it at all is strong evidence it really happened as depicted.
How exactly would the government ban training runs? Like how would that work? What would the legislation text look like? And how would it be enforceable without running afoul of the vagueness doctrine?
2 replies →
When the bubble pops, their customer is the military/government, and the goal is regulation. "Our thing did another impressive hack" helps both of those goals.
Not saying it's intentional, but it does move them where they wanna go.
I think a lot of people here love AI and don't want it taken away. Calling every red flag marketing is a way to trivialize the problem and prevent regulation.
When you understand people's motives, then it's easier to understand why they call it marketing. Personally I feel like it's almost an unconscious reaction to cope with the fact that this stuff is getting really dangerous and they can't believe it themselves - stick your head in the sand and call it marketing.
5 replies →
The relevant "customers" are investors, not "the average consumer"