Comment by dougb5
2 years ago
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.
'Fearing a rise of killer robots is like worrying about overpopulation on Mars.' - Andrew Ng
https://www.wired.com/brandlab/2015/05/andrew-ng-deep-learni... (2015)
> What’s the most valid reason that we should be worried about destructive artificial intelligence?
> I think that hundreds of years from now if people invent a technology that we haven’t heard of yet, maybe a computer could turn evil. But the future is so uncertain. I don’t know what’s going to happen five years from now. The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars. Hundreds of years from now I hope we’ve colonized Mars. But we’ve never set foot on the planet so how can we productively worry about this problem now?
Well, to steelman the ‘overpopulation on Mars’ argument a bit, feeding 4 colonists and feeding 8 is a 100% increase in food expenditure, which may or may not be possible over there. It might be courtains for a few of them if it comes to that.
1 reply →
Sentient killer robots is not the risk most AI researchers are worried about. The risk is what happens as corporations give AI ever larger power over significant infrastructure and marketing decisions.
Facebook is an example of AI in it's current form already doing massive societal damage. It's algorithms optimize for "success metrics" with minimal regard for consequences. What happens when these algorithms are significantly more self modifying? What if a marketing campaign realizes a societal movement threatens it's success? Are we prepared to weather a propaganda campaign that understands our impulses better than we ever could?
This might have to bump out "AI is no match for HI (human idiocy)" as the pithy grumpy old man quote I trot out when I hear irrational exuberance about AI these days.
At the current Mars’ carrying capacity, one single person could be considered an overpopulation problem.
Unfortunately, robots that kill people already exist. See: semi-autonomous war drones
Andrew Ng worked on facial recognition for a company with deep ties to the Chinese Communist Party. He’s the absolute worst person to quote.
1 reply →
So, this is actually an aspect of superintelligence that makes it way more dangerous than most people think. That we have no way to know if any given alignment technique works for the N+1 generation of AIs.
It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.
Fortunately, whenever you create a superintelligence, you obviously have a choice as to whether you confine it to inside a computer or whether you immediately hook it up to mobile robots with arms and fine finger control. One of these is obviously the far wiser choice.
As long as you can just turn it off by cutting the power, and you're not trying to put it inside of self-powered self-replicating robots, it doesn't seem like anything to worry about particularly.
A physical on/off switch is a pretty powerful safeguard.
(And even if you want to start talking about AI-powered weapons, that still requires humans to manufacture explosives etc. We're already seeing what drone technology is doing in Ukraine, and it isn't leading to any kind of massive advantage -- more than anything, it's contributing to the stalemate.)
9 replies →
“it is difficult to get a man to understand something, when his salary depends on his not understanding it.” - Upton Sinclair
The counter argument is viewing it like nuclear energy. Even if its in the early days of our understanding of nuclear energy, seems pretty good to have a group working towards creating safe nuclear reactors, vs just trying to create nuclear reactors
Nuclear energy was at inception and remains today wildly regulated, in generally (outside of military contexts) a very transparent way, and the brakes get slammed on over even minor incidents.
It’s also of obvious as opposed to conjectural utility: we know exactly how we price electricity. There’s no way to know how useful a 10x large model will be, we’re debating the utility of the ones that do exist, the debate about the ones that don’t is on a very slender limb.
Combine that with a political and regulatory climate that seems to have a neon sign on top, “LAWS4CA$H” and helm the thing mostly with people who, uh, lean authoritarian, and the remaining similarities to useful public projects like nuclear seems to reduce to “really expensive, technically complicated, and seems kinda dangerous”.
Folks understood the nuclear forces and the implications and then built a weapon using that knowledge. These guys don't know how to build AGI and don't have the same theoretical understanding of the problem at hand.
Put another way, they understood the theory and applied it. There is no theory here, it's alchemy. That doesn't mean they can't make progress (the progress thus far is amazing) but it's a terrible analogy.
It would be akin to creating a "safe Dyson sphere", though; that's all it is.
If your hypothetical Dyson sphere (WIP) has a big chance to bring a lot of harm, why build it in the first place?
I think the whole safety proposal should be thought of from that point of view. "How do we make <thing> more beneficial than detrimental for humans?"
Congrats, Ilya. Eager to see what comes out of SSI.
InstructGPT is basically click through rate optimization. The underlying models are in fact very impressive and very capable for a computer program, but they’re then subject to training and tuning with the explicit loss function of manipulating what human scorers click on, in a web browser or the like.
Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.
You think we should try to create an unsafe Dyson Sphere first? I don't think that's how engineering works.
I think it’s clear we are at least at the remotely resembling intelligence stage… idk seems to me like lots of people in denial.
To a technoutopian, scientific advances, and AI in particular, will one day solve all other human problems, create heaven on earth, and may even grant us eternal life. It's the most important problem in the same way that Christ's second coming is important in the Christian religion.
I had a very smart tech person tell me at a scientific conference a few weeks ago, when I asked "why do we want to create AGI in the first place", that AGI could solve a host of human problems, including poverty, hunger. Basically, utopia.
I was quite surprised at the naiveté of the answer given that many of these seemingly intractable problems, such as poverty, are social and political in nature and not ones that will be solved with technology.
Update: Even say a super AI was able to figure out something like cold fusion thereby "solving" the energy problem. There are so many trillions of dollars of vested interests stacked against "free clean energy for all" that it would be very very difficult for it to ever see the light of day. We can't even wean ourselves off coal for crying out loud.
It’s amazing how someone so smart can be so naive. I do understand conceptually the idea that if we create intelligence greater than our own that we could struggle to control it.
But does anyone have any meaningful thoughts on how this plays out? I hear our industry thought leaders clamoring over this but not a single actual concrete idea of what this means in practice. We have no idea what the fundamental architecture for superintelligence would even begin to look like.
Not to mention the very real counter argument of “if it’s truly smarter than you it will always be one step ahead of you”. So you can think you have safety in place but you don’t. All of your indicators can show it’s safe. Every integration test can pass. But if you were to create a superintelligence with volition, you will truly never be able to control it, short of pulling the plug.
Even more so, let’s say you do create a safe superintelligence. There isn’t going to be just one instance. Someone else will do the same, but make it either intentionally unsafe or incidentally through lack of controls. And then all your effort is academic at best if unsafe superintelligence really does mean doomsday.
But again, we’re far from this being a reality that it’s wacky to act as if there’s a real problem space at hand.
While the topic of "safe reasoning" may seem more or less preliminary before a good implementation of reasoning, it remains a theoretical discipline with its own importance and should be studied alongside the rest, also largely irregardless if its stage.
> We have no idea what the fundamental architecture for superintelligence would even begin to look like
Ambiguous expression. Not implemented technically does not mean we would not know what to implement.
You’re assuming a threat model where the AI has goals and motivations that are unpredictable and therefore risky, which is certainly the one that gets a lot of attention. But even if the AI’s goals and motivations can be perfectly controlled by its creators, you’re still at the mercy of the people who created the AI. In that respect it’s more of an arms race. And like many arms races, the goal might not necessarily be to outcompete everyone else so much as maintain a balance of power.
There’s no safe intelligence, so there’s no safe superintelligence. If you want safer superintelligence, you figure out how to augment the safest intelligence.
See MIRI https://intelligence.org/
"how someone so smart can be so naive"
Do you really think Ilya has not thought deeply about each and every one of your points here? There's plenty of answers to your criticisms if you look around instead of attacking.
I actually do think they have not thought deeply about it or are willfully ignoring the very obvious conclusions to their line of thinking.
Ilya has an exceptional ability extrapolate into the future from current technology. Their assessment of the eventual significance of AI is likely very correct. They should then understand that there will not be universal governance of AI. It’s not a nuclear bomb. It doesn’t rely on controlled access to difficult to acquire materials. It is information. It cannot be controlled forever. It will not be limited to nation states, but deployed - easily - by corporations, political action groups, governments, and terrorist groups alike.
If Ilya wants to make something that is guaranteed to avoid say curse words and be incapable of generating porn, then sure. They can probably achieve that. But there is this naive, and in all honesty, deceptive, framing that any amount of research, effort, or regulation will establish an airtight seal to prevent AI for being used in incredibly malicious ways.
Most of all because the most likely and fundamentally disruptive near term weaponization of AI is going to be amplification of disinformation campaigns - and it will be incredibly effective. You don’t need to build a bomb to dismantle democracy. You can simply convince its populace to install an autocrat favorable to your cause.
It is as naive as it gets. Ilya is an academic and sees a very real and very challenging academic problem, but all conversations in this space ignore the reality that knowledge of how to build AI safely will be very intentionally disregarded by those with an incentive to build AI unsafely.
4 replies →
I mean if you just take the words on that website at face value, it certainly feels naive to talk about it as "the most important technical problem of our time" (compared to applying technology to solving climate change, world hunger, or energy scarcity, to name a few that I personally think are more important).
But it's also a worst-case interpretation of motives and intent.
If you take that webpage for what it is - a marketing pitch - then it's fine.
Companies use superlatives all the time when they're looking to generate buzz and attract talent.
1 reply →
> There isn’t going to be just one instance. Someone else will do the same
NK AI (!)
We're really not that far. I'd argue superintelligence has already been achieved, and it's perfectly and knowably safe.
Consider, GPT-4o or Claude are:
• Way faster thinkers, readers, writers and computer operators than humans are
• Way better educated
• Way better at drawing/painting
... and yet, appear to be perfectly safe because they lack agency. There's just no evidence at all that they're dangerous.
Why isn't this an example of safe superintelligence? Why do people insist on defining intelligence in only one rather vague dimension (being able to make cunning plans).
Yann LeCun said it best in an interview with Lex Friedman.
LLMs don't consume more energy when answering more complex questions. That means there's no inherent understanding of questions.
(which you could infer from their structure: LLMs recursively predict the next word, possibly using words they just predicted, and so on).
5 replies →
> Way faster thinkers, readers, writers and computer operators than humans are
> Way better educated
> Way better at drawing/painting
I mean this nicely, but you have fallen for the anthropomorphizing of LLMs by marketing teams.
None of this is "intelligent", rather it's an incredibly sophisticated (and absolutely beyond human capabilities) lookup and classification of existing information.
And I am not arguing that this has no value, it has tremendous value, but it's not superintelligence in any sense.
LLMs do not "think".
1 reply →
This all makes more sense when you realise it's Calvinism for programmers.
I think I heard that one before. Nuclear weapons are the Armageddon of nerds. Climate change is the Flood of the nerds. And so on.
Could you expand on this?
> [Superintelligence safety] teaches that the glory and sovereignty of [superintelligence] should come first in all things.
1 reply →
Calvinism for Transcendentalist techno-utopians -- an Asimovian Reformation of Singulatarianism
I think the idea is that a safe super intelligence would help solve those problems. I am skeptical because the vast majority are social coordination problems, and I don’t see how a machine intelligence no matter how smart can help with that.
So instead of a super intelligence either killing us all or saving us from ourselves, we’ll just have one that can be controlled to extract more wealth from us.
IMO, this is the most likely outcome
Social coordination problems exist within a specific set of constraints, and that set of constraints can itself be altered. For instance, climate change is often treated as a social coordination problem, but if you could produce enough energy cheaply enough, you could solve the greenhouse gas problem unilaterally.
OK, lets play this out.
Lets say an AI discovers cold fusion. Given the fact that it would threaten to render extinct one of the largest global economic sectors (oil/gas), how long do you think it would take for it to actually see the light of day? We can't even wean ourselves off coal.
3 replies →
I largely agree, although I do see how AI can help with social coordination problems, for example by helping elected leaders be more responsive to what their constituents need. (I spend a lot of my own time working with researchers at that intersection.) But social coordination benefits from energy research, too, and from biology research, and from the humanities, and from the arts. Computer science can't singlehandedly "solve" these problems any more than the other fields can; they are needed together, hence my gripe about total-orderings.
Exactly. Or who gets the results of its outputs. How do we prioritize limited compute?
Even not just the compute but energy use at all. All the energy burned on training just to ask it the stupidest questions, by the numbers at least. All that energy that could have been used to power towns, schools, and hospitals the world over that lack sufficient power even in this modern age. Sure there's costs to bringing power to someplace, its not handwavy but a hard problem, but still, it is pretty perverse where our priorities lie in terms of distributing the earths resources to the earths humans.
3 replies →
are humans smarter than apes, and do humans do a better job at solving social coordination problems?
> I am skeptical because the vast majority are social coordination problems, and I don’t see how
Leadership.
By any means necessary I presume. If Russian propaganda helped get Trump elected, AI propaganda could help social coordination by influencing public perception of issues and microtargeting down to the individual level to get people on board.
could but it's owners might have a vested interest in influencing public perceptions to PREVENT positive social outcomes and favor the owners financial interests.
(seems rather more likely, given who will/would own such a machine)
It says «technical» problem, and probably implies that other technical problems could dramatically benefit from such achievement.
If you want a real technical revolution, you teach the masses how to code their own tailored software, and not just use abstractions and software built by people who sell software to the average user. What a shame we failed at that and are even sliding back in a lot of ways with plummeting technical literacy in smartphone-raised generations.
> you teach the masses how to code their own tailored software
That does not seem to be the key recipe to reaching techno-scientific milestones - coders are not necessarily researchers.
> plummeting technical literacy in smartphone-raised generations
Which shows there are other roots to the problem, given that some of us (many probably in this "club") used our devices generally more productively than said «generations»... Maybe it was a matter of will and education? Its crucial sides not being «teach[ing] the masses how to code»...
1 reply →
this.
The blanket statements on the SSI homepage are pretty mediocre, and it is only the reputation of the founders that carries the announcement.
I think this quote at the end of this Bloomberg piece[0] gives more context,
> Sutskever says that the large language models that have dominated AI will play an important role within Safe Superintelligence but that it’s aiming for something far more powerful. With current systems, he says, “you talk to it, you have a conversation, and you’re done.” The system he wants to pursue would be more general-purpose and expansive in its abilities. “You’re talking about a giant super data center that’s autonomously developing technology. That’s crazy, right? It’s the safety of that that we want to contribute to.”
[0]: https://www.bloomberg.com/news/articles/2024-06-19/openai-co...
[0]: https://archive.is/ziMOD
So you're surprised when someone admits choosing to work on the problem they believe is the biggest and most important?
I guess they could be lying or badly disconnected from reality as you suggest. It would be far more interesting to read an argument for another problem being more valuable. It would be far cooler to hear about a plausible solution you're working on to solve that problem.
Yes, they see it as the top problem, by a large margin.
If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"
> the superlatives in these sorts of announcements really annoy me
I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.
Technical. He's saying it's the most important technical problem of our time.
Basically every problem is a "technical" problem in the year 2024 though? What problems out there don't have a solution that leverages technology?
>What problems out there don't have a solution that leverages technology?
Societal problems created by technology?
1 reply →
[dead]
It's Palo Alto & Tel Aviv ordering that is total.
You don't need to survey every problem to feel some problem might be the most important one. If you think AGI/ASI is coming soon and extinction risks are high, you don't really need to order to see it's the most important problem.
It certainly is the most important technical problem of our time, if we end up developing such a system.
That conditional makes all the difference.
It's a hell of a conditional, though.
"How are all those monkeys flying out of my butt?" would be the important technical problem of our time, if and only if, monkeys were flying out of my butt.
It's still not a very important statement, if you downplay or omit the conditional.
Is "building safe superintelligence (SSI) is the most important technical problem of our time" full stop ?
Is it fuck.
Yeah — that was exactly my (slightly sarcastic) point.
Let us know if you ever encounter that monkey problem, though. Hopefully we can all pull together to find a solution.
1 reply →
[dead]
It is the most important problem of “our time” when you realize that the “our” here has the same meaning that it has in “our democracy”
C'mon. This one-pager is a recruiting document. One wants 'true believers' (intrinsically motivated) employees to execute the mission. Give Ilya some slack here.
Fair enough, and it's not worse than a lot of other product marketing messages about AI these days. But you can be intrinsically motivated by a problem without believing that other problems are somehow less important than yours.
exactly. and define safe. eg, is it safe (ie dereliction) to _not_ use ai to monitor dirty bomb threats? or more simple, CSAM?
In the context of super-intelligence, “safe” has been perfectly well defined for decades: “won't ultimately result in everyone dying or worse”.
You can call it hubris if you like, but don't pretend like it's not clear.
It’s not, when most discussion around AI safety in the last few years has boiled down to “we need to make sure LLMs never respond with anything that a stereotypical Berkeley progressive could find offensive”.
So when you switch gears and start using safety properly, it would be nice to have that clarified.
Love to see the traditional middlebrow dismissal as the top comment. Never change, HN.
> Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
It might be the case that the reason Ilya is “interested in” this problem (to the degree of dedicating almost his entire career to it) is exactly because he believes it’s the most important.
This. Next will hyperintelligence(R) /s
I believe that AGI is the last problem in computer science, so solving it solves all of the others. Then with AGI, we can solve the last remaining problems in physics (like unifying gravity with quantum mechanics), biology (administering gene therapy and curing death), etc.
But I do agree that innovations in tech are doing little or nothing to solve mass suffering. We had the tech to feed everyone in the world through farm automation by the 60s but chose not to. We had the tech in the 80s to do moonshots for AIDS, cancer, etc but chose not to. We had the tech in the 2000s to transition from fossil fuels to renewables but chose not to. Today we have the opportunity to promote world peace over continuous war but will choose not to.
It's to the point where I wonder how far innovations in tech and increases in economic productivity will get without helping people directly. My experience has been that the world chooses models like Dubai, Mexico City and San Francisco where skyscrapers tower over a surrounding homeless and indigent population. As long as we continue pursuing top-down leadership from governments and corporations, we'll see no change to the status quo, and even trends towards authoritarianism and fascism. It will take people at the bottom organizing to provide an alternate economic model before we have options like universal education/healthcare/opportunity and UBI from robot labor.
What gets me is that stuff like the ARC prize for AGI will "just work". As in, even if I had a modest stipend of a few thousand dollars per month to dabble in AI and come up with solutions the way I would for any other startup, certainly within 3 years, someone else would beat me to it. There simply isn't enough time now to beat the competition. Which is why I give AGI over 50% odds of arriving before 2030, where I used to think it was 2040 or 2050. The only thing that could stop it now is sabotage in the form of another global pandemic, economic depression or WWIII. Progress which threatens the power structures of the ultra wealthy is what drives the suffering that they allow to continue.