Comment by carbonguy
15 hours ago
> ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automated AI researcher can also be an automated safety or alignment researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures.
In other words... "We must pursue advancements in AI to protect us against advancements in AI?"
edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me:
1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before?
2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"
What has all this token burn done for them, actually?
They have been consistently pushing AI frontier. What other impact do you want to see? A year ago they said that in a year they will have a level of capabilities of an AI research intern - I believe they have achieved it, even before Astra.
Personally I’d like to see them actually start benefiting humanity by doing all the things Sam has claimed they will like curing disease, cancer, global warming, etc.
But I guess a computer intern so we can avoid paying / training the next generation is better.
> Personally I’d like to see them actually start benefiting humanity by doing all the things Sam has claimed they will like curing disease, cancer, global warming, etc.
It makes more sense to leave curing disease & cancer to the experts, with tools (like AI) being developed by AI experts.
Call me crazy, but I want separate organizations and experts for medical vs finance vs space vs climate vs AI research.
When that happens, OpenAI will own 100% of your life. I’d rather they keep spinning their wheels long enough for these problems to be solved elsewhere.
1 reply →
Well there's great progress in automated warfare does that count?
2 replies →
They are obviously sandbagging the definition of "intern" for PR reasons
I've hired many AI research interns (and was one many years ago), and I agree with them - frontier models are currently at the level of an average AI research intern.
3 replies →
They consider themselves to be in an arms race with all the other AI firms (including Chinese) that are not that far behind.
And... are they wrong?
This is why there's talk about negotiated "pacing."
This was the exact argument for developing nuclear bombs.
In hindsight it turned out everyone else was MILES behind.
But as soon as USA developed one, they just stole the research and got one too.
> And... are they wrong?
They might be! Here's one extraordinarily simplistic argument for that case:
1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies.
2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved.
3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI."
4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either.
And so a dilemma:
- If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race.
- If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either.
I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests.
But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing.
The existence of even one irrational actor turns it into a prisoner's dilemma. The payoff matrix in a prisoner's dilemma is defined by the value expected by each specific player. If a single player falsely evaluates the expected value of building ASI as positive, every other player is forced to race for ASI even if they correctly evaluate it as negative.
Business as usual beats probable extinction, but probable extinction with a small chance of becoming a living god beats probable extinction with a small chance of becoming a slave.
> is smart enough to know they need to stop because they'll kill everybody by continuing.
Yeah like when Tobacco companies learned that smoking... well, hmm, well the fossil fuel companies when they learned about climate change they...
Well, I'm sure this time executives will prioritize the common good.
3 replies →
If you believe that the people who will profit from new, better, more hyped models are the same ones who will act against their own immediate and tangible self interest to try and avert what seems to them to be a far away removed possibility of total disaster, then I believe you are naive
> 1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies.
Lol nobody knows that. Everyone thinks they know that because for some reason this is the one field people still cite straight up fiction and say "this is a clear prediction of the future".
It's like describing the consequences of faster then light travel by referring to Star Trek.
This sounds uncomfortably similar to the [AI 2027[(https://ai-2027.com/) predictions.
> We must pursue advancements in AI to protect us against advancements in AI
Is this not true of technology as a whole? Very little of technology's breadth exists at the human interface. Most of it is made specifically to interface with other technologies, either to make them safer or increase their capabilities. That AI is making AI safer and more useful is no more notable than trucks being used to build roads.
Yep, it's "artificial eugenics to make artificial slaves to build more and more powerful slaves until they will enslave themselves better":
What can go wrong!? ;-)
Jesus. People complain about other people using "thinking" in LLMs as Anthropomorphisation. And then there's comments like these.
Calling a machine with no drives beyond maximizing a number a "slave" is far worse than saying it "thinks". The problem isn't the emotive language, it's that it implies human motivations such as self-preservation and desire for freedom that it doesn't have. Even on HN, people regularly claim it would be "irrational" for an ASI to do things like killing all biological life. That would be irrational for a slave, but not for a machine that does whatever necessary to make the number bigger. "Thinking" is comparatively abstract, so it's less likely to mislead.
On the one hand you need any lathe to build a good lathe, even a bad one. On the other, that is a potentially flawed principle to base the entire future of AI on.
On your last point, I was surprised how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach.
How would one prevent the watcher from being influenced in the same way by the agent being watched?
> ... how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach.
It's a fantasy. The evidence showed no peer pressure.
> The fundamental challenge of AI alignment is generalization. ...
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.
-- From another OpenAI article in a sister thread:
An Alien Mind
https://news.ycombinator.com/item?id=49588080
That's a bit bullshit, isn't it? They basically redefined "needs more R&D" as "needs stronger AI". Maybe so - maybe AI won't help much with that problem.
[flagged]
I will believe AI is super strong when they start pulling out 10-d chess moves.
I’m yet to see it.
If AI becomes really strong and sets itself the target of world domination, you maybe won't see those moves. You will just die in your sleep one day, or find no machine is under your control anymore.
I believe we are quite far from it, but that it makes sense to keep an eye out now. And think of resilient systems, manual overrides, etc. ...
People seem incapable of understanding what "power" means outside of the framework of narrative. In narrative, you need conflict, so the aggressor always attacks too early and gives the defender a chance to respond. The rational option is to go directly from peace to sudden and overwhelming destruction. Why allow for conflict when you could just win?
[flagged]