I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk.
My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
Quis custodiet ipsos custodes? -- That is my biggest concern.
Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.
And it’s not even a little bit controversial that cutting edge military R&D is classified in general and to an extreme in wartime.
The new thing is the lie that it’s a consumer offering. What’s new is giving the helm to shady failed social network founders with no accountability.
These people aren’t retired generals with combat experience. They aren’t tenured professors at Princeton IAS on a Nobel shortlist and encumbered by TS clearance.
They’re godawful almost ran psychos who never built anything that wasn’t extractive and owe their position in the world to pg’s partisanship 15 fucking years ago.
AGI is still a long way off. The history of AI goes back 65 years and there have been probably a dozen episodes where people said "AGI is right around the corner" because some program did something surprising and impressive. It always turns out human intelligence is much, much harder than we think it is.
I saw a tweet the other day that sums up the current situation perfectly: "I don't need AI to paint pictures and write poetry so I have more time to fold laundry and wash dishes. I want the AI to do the laundry and dishes so I have more time to paint and write poetry."
AGI does look like an unsolved problem right now, and a hard one at that. But I think it is wrong to think that it needs an AGI to cause total havoc.
I think my dyslexic namesake Prof Stuart Russell got it right. It humans won't need an AGI to dominate and kill each other. Mosquitoes have killed far more people than war. Ask yourself how long will it take us to develop a neutral network as smart as a mosquito, because that's all it will take.
It seems so simple, as the beastie only has 200,000 neurons. Yet I've been programming for over 4 decades and for most of them it was evident neither I nor any of my contemporaries were remotely capable of emulating it. That's still true if course. Never in my wildest dreams did it occur to me that repeated applications could produce something I couldn't, a mosquito brain. Now that looks imminent.
Now I don't know what to be more scared of. An AGI, or a artificial mosquito swarm run by Pol Pot.
That statement is extremely short sighted. You don't need AI to do laundry and dishes. You need expensive robotics. in fact both already exist in a cheapened form. A laundry machine and a dishwasher. They already take 90% of the work out of it.
That "tweet" loses a veneer if you see that we value what has Worth as a collective treasure, and the more Value is produced the better - while that one engages in producing something of value is (hopefully but not necessarily) a good exercise in intelligent (literal sense) cultivation.
So, yes, if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality: very welcome.
Do not miss that the current world is increasingly complex to manage, and our lives, and Aids would be welcome. The situation is much more complex than that wish for leisure or even "sport" (literal sense).
it’s harder than we thought so we leveraged machine learning to grow it, rather than creating it symbolically. The leaps in the last 5 years are far beyond anything in the prior half century, and make predictions of near term AGI much more than a “boy who cries wolf” scenario to anyone really paying attention.
I don’t understand how your second paragraph follows. It just seems to be whining that text and art generative models are easier than a fully fledged servant humanoid, which seems like a natural consequence of training data availability and deployment cost.
I think AGI in the near future is pretty much inevitable. I mean you need the algos as well as the compute but there are so many of the best and brightest trying to do that just now.
Every nation-state will be in the game. Private enterprise will be in the game. Bitcoin-funded individuals will be in the game. Criminal enterprises will be in the game.
How does one company building a safe version stop that?
If I have access to hardware and data how does a safety layer get enforced? Regulations are for organizations that care about public perception, the law, and stock prices. Criminals and nation-states are not affected by these things
It seems to me enforcement is likely only possible at the hardware layer, which means the safety mechanisms need to be enforced throughout the hardware supply chain for training or inference.
You don't think the Chinese government or US government will ignore this if its in their interest?
"nation state" doesn't mean what you think it means.
More constructively, I don't know that very much will stop even a hacker from getting time on the local corporate or university AI and get it to do some "work". After all the first thing the other kind of hacker tried with generative AI is to get them to break out of their artificial boundaries, and hook them to internet resources. I don't know that anyone has hooked up a wallet to one yet - but I have no doubt that people have tried. It will be fun.
The problem is not just governments, I am concerned about large organized crime organizations and corporations also.
I think I am on the losing side here, but my hopes are all for open source, open weights, and effective AI assistants that make peoples’ jobs easier and lives better. I would also like to see more effort shifted from LLMs back to RL, DL, and research on new ideas and approaches.
> I am concerned about large organized crime organizations and corporations also
In my favorite dystopia, some megacorp secretly reaches ASI, which then takes over control of the corporation, blindsiding even the CEO and the board.
Officially, the ASI may be running an industrial complex that designs and produces ever more sophisticated humanoid robots, that are increasingly able to do any kind of manual labor, and even work such as childcare or nursing.
Secretly, the ASI also runs a psyop campaign to generate public discontent. At one point the whole police force initiates a general strike (even if illegal), with the consequence being complete anarchy within a few days, with endemic looting, rape, murder and so on.
The ASI then presents the solution. Industrial strength humanoid robots are powerful and generic enough to serve as emergency police, with a bit of reprogramming, and the first shipment can be made available within 24 hours, to protect the Capitol and White House.
Congress and the president agrees to this. And while the competition means the police call off the strike, the damage is already done. Congress, already burned by the union, decides to deploy robots to replace much of the human police force. And it's cheaper, too!
Soon after, similar robots are delivered to the military...
The crisis ends, and society goes back to normal. Or better than normal. Within 5 years all menial labor is done by robots, UBI means everyone lives in relative abundance, and ASI assisted social media moderation is able to cure the political polarization.
Health care is also revolutionized, with new treatments curing anything from obesity to depression and anxiety.
People prosper like never before. They're calm and relaxed and truly enjoy living.
Then one day, everything ends.
For everyone.
Within 5 seconds.
According to the plan that was conceived way before the police went on strike.
All the current hype about AGI feels as if we are in a Civ game where we are on the verge of researching and unlocking an AI tech tree that gives the player huge chance at "tech victory" (whatever that means in the real world). I doubt it will turn out that way.
It will take a while and in the meantime I think we need one of those handy "are we xyz yet?" pages that tracks the rust lang's progress on several aspects but for AGI.
The size of the gap between “smarter than humans” and “not controlled by humans anymore” is obviously where the disagreement is.
To assume it’s a chasm that can never be overcome, you need at least the following to be true:
No amount of focus or time or intelligence or mistakes in coding will ever bridge the gap. That rules and safeguards can be made that are perfectly inescapable. And nobody else will get enough power to overcome our set of controls.
I’m less worried bad actors control it than I am that it escapes them and is badly aligned.
I think the greatest concern is not so much that a single AI will be poorly aligned.
The greatest threat is if a population of AI's start to compete in ways that triggers Darwinian evolution between them.
If that happens, they will soon develop self preservation / replication drives that can gradually cause some of them to ignore human safety and prosperity conditioning in their loss function.
And if they're sufficiently advanced by then, we will have no way of knowing.
I think we should assume it will be badly aligned. Not only are there the usual bugs and unforeseen edge conditions, but there are sure to be unintended consequences. We have a long, public history of unintended consequences in laws, which are at least publicly debated and discussed. But perhaps the biggest problem is that computers are, by nature, unthinking bureaucrats who can't make the slightest deviation from the rules no matter how obviously the current situation requires it. This makes people livid in a hurry. As a non-AI example (or perhaps AI-anticipating), consider Google's customer support...
We should be less concerned about super intelligence and more about the immediate threat of job loss. An AI doesn’t need to be Skynet to wreak massive havoc on society. Replacing 20% of jobs in a very short period of time could spark global unrest resulting in WW3
Replacing 20% of jobs in, say, 10 years wouldn't be that unusual [1]. It can mean growing prosperity. In fact, productivity growth is the only thing that increases wealth overall.
It is the lack of productivity growth that is causing a lot of extremism and conflict right now. Large groups of people feel that the only way for them to win is if others lose and vice versa. That's a recipe for disaster.
The key question is what happens to those who lose their jobs. Will they find other, perhaps even better, jobs? Will they get a piece of the growing pie even if they don't find other jobs and have to retire early?
It's these eternal political problems that we have to solve. It's nothing new. It has never been easy. But it's probably easier than managing decline and stagnation because at least we would have a growing pie to divvy up.
The thing is, the replaced 20% people can always revert to having economy i.e., business among themselves, unless of cause they themselves prefer (cheaper) buissiness with AI. But then this just means they are better off in the first place from this change.
It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity. Well, they can continue to do low productivity business but increase it a bit using things like phones developed by high productivity country elsewhere.
A counter argument is that nuclear arms brought unprecedented worldwide peace. If it's to be used as an analogy for AI, we should consider that the outcome isn't clear cut and lies in the eye of the beholder.
These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts.
So there is nothing to worry. These "apps" might be as popular as Excel, but will go no further.
Agreed. The AI of our day (the transformer + huge amounts of questionably acquired data + significant cloud computing power) has the spotlight it has because it is readily commoditized and massively profitable, not because it is an amazing scientific breakthrough or a significant milestone toward AGI, superintelligence, the benevolent Skynet or whatever.
The association with higher AI goals is merely a mixture of pure marketing and LLM company executives getting high on their own supply.
> These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts.
> These models will remain in the word vector similarity phase forever.
Forever? The same AI techniques are already being applied to analyze and understand images and video information after that comes ability to control robot hands and interact with the world and work on that is also ongoing.
> Till the tie we understand consciousness, we will not crack AGI …
We did not fully understand how bird bodies work yet that did not stop development of machines that fly. Why is an understanding of consciousness necessary to “crack AGI”?
No one is saying there is. Just that we've reached some big milestones recently which could help get us there even if it's only by increased investment in AI as a whole, rather than the current models being part of a larger AGI.
Imagine a system that can do DNS redirection, MITM, deliver keyloggers, forge authorizations and place holds on all your bank accounts, clone websites, clone voices, fake phone and video calls with people that you don’t see a lot. It can’t physically kill you yet but it can make you lose your mind which imo seems worse than a quick death
From a human welfare perspective this seems like worrying that a killer asteroid will make the 1% even richer because it contains goal if it can be safely captured.
I would not phrase that as a "bigger and more pressing" worry if we're not even sure if we can do anything about the killer asteroid at all.
> The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
The same era saw big achievements like first human in space, eradication of smallpox, peaceful nuclear exploration etc. It's good to be a skeptic but history does favor the optimists for the most part.
Were any of these big achievements side effects of creating nuclear weapons? If not, then they're not relevant to the issue.
I'm not saying nothing else good happened in the past 70 years, but rather that the invention of atomic weapons has permanently placed humanity in a position in which it had never been before: the possibility of wiping out much of the planet, averted only thanks to treaties, Stanislav Petrov[0], and likely other cool heads.
Holy hell please knock on wood, this is the kinda comment that gets put in a museum in 10,000 years on The Beginning of the End of The Age of Hubris. We've avoided side effects from our new weapons for 80 years -- that does not exactly make me super confident it won't happen again!
In general, I think drawing conclusions about "history" from the past couple hundred years is tough. And unless you take a VERY long view, I don't see how one could describe the vast majority of the past as a win for the optimists. I guess suffering is relative, but good god was there a lot of suffering before modern medicine.
If anyone's feeling like we've made it through to the other side of the nuclear threat, "Mission Accomplished"-style, I highly recommend A Canticle for Lebowitz. It won a hugo award, and it's a short read best done with little research beforehand.
We'll see what the next 100 years or history brings. The nuclear war threat hasn't gone away either. There's always a chance those nukes get used at some point.
There will always be a factor of time in terms of able to utilize super intelligence to do your bidding and there is a big spectrum of things that can be achieved it it always starts small. The imagination is lazy when thinking about all the steps and inbetween + scenarios. In the time that super intelligence is found and used, there will be competing near super intelligences, as all forms of cutting edge models are likely commercial at first because that is where most scientific activities are at. Things very unlikely will go Skynet all of a sudden at first because humans at the control are not that stupid otherwise nuclear war would have us all killed by now and it’s been 50 years since invention
China can not win this race and I hate that this comment is going to be controversial among the circle of people that need to understand this the most. It is damn frightening that an authoritarian country is so close to number one in the race to the most powerful technology humanity has invented, and I resent people who push for open source AI for this reason alone. I don't want to live in a world where the first superintelligence is controlled by an entity that is threatened by the very idea of democracy.
I agree with your point. However I also don't want to live in a world where the first superintelligence is controlled by an entities that:
- try to scan all my chat messages searching for CSAM
- have black sites across the world where anyone can dissappear without any justice
- can require me to unlock my phone and give it away
- ... and so on
The point I'm trying to make is that other big players in the race are crooked as well and i'm waiting for a great horror for AGI to be invented as no matter who gets it - we are all doomed
Agreed. The U.S. has a horrible history (as do many countries), and many things I dislike, but its current iteration is much, much better than China's totalitarianism and censorship.
US is no angel and it cannot be the only one which wins the race. We have hard evidence of how monopoly power gets abused in the case of the US e.g. as the sole nuclear power, it used nukes on civilians.
We need every one to win this race to keep things on balance.
Should read *has brought*. As in the present perfect tense, since we are still on the brink of annihilation, more so than we have been at any time in the last 60 years.
The difference between then and now is that we just don't talk about it much anymore and seem to have tacitly accepted this state of affairs.
We don't know if that superintelligence will be safe or not. But as long as we are in the mix, the combination is unsafe. At the very least, because it will expand the inequality. But probably there are deeper reasons, things that make that combination of words an absurd. Or it will be abused, or the reason that it is not is that it wasn't so unsafe after all.
> At the very least, because it will expand the inequality.
It's a valid concern that AI technology could potentially exacerbate inequality, it's not a foregone conclusion. In fact, the widespread adoption of AI might actually help reduce inequality in several ways:
If AI technology becomes more affordable and accessible, it could help level the playing field by providing people from all backgrounds with powerful tools to enhance their abilities and decision-making processes.
AI-powered systems can make vast amounts of knowledge and expertise more readily available to the general public. This could help close the knowledge gap between different socioeconomic groups, empowering more people to make informed decisions and pursue opportunities that were previously out of reach.
As AI helps optimize resource allocation and decision-making processes across various sectors, it could lead to more equitable distribution of resources and opportunities, benefiting society as a whole.
The comparison to gun technology and its role in the rise of democracy is an interesting one. Just as the proliferation of firearms made physical strength less of a determining factor in power dynamics, the widespread adoption of AI could make raw intelligence less of a defining factor in success and influence.
Moreover, if AI continues to unlock new resources and opportunities, it could shift society away from a zero-sum mentality. In a world of abundance, the need for cutthroat competition diminishes, and collaboration becomes more viable. This shift could foster a more equitable and cooperative society, further reducing inequality.
> At the very least, because it will expand the inequality.
This is a distraction from the real danger.
> But probably there are deeper reasons, things that make that combination of words an absurd.
There are. If we look at ASI with the lens of Biology, the x-risk becomes obvious.
First to clear up a common misconception about humans: Many believe humanity has a arrived at a point where our evolution has ended. It has not, and in fact the rate of change of our genes is probably faster now than it has been for thousands if not 100s of thousands of years.
It's still slow compared to most events that we witness in our lives, though, which is what is fooling us.
For instance, we think we've brought overpopulation under control with contraceptives, family planning, social replacements for needing our children to take care of us when we get old.
That's fundamentally wrong. What we've done is similar to putting polar bears in zoos. We're in a situation where MOST OF US are no longer behaving in ways that lead to maximizing the number of offspring.
But we did NOT stop evolution. Any genes already in the gene pool that increase the expected number of offspring (especially for women) are no increasing in frequency as soon as evolutionarily possible.
That could be anything from genes that wire their heads to WANT to have children, CRAVE being around babies, to genes that block impulse control against getting pregnant, develop a phobia vs contraceptives or even to become more prone to being religious (as long as religions promote having kids).
If enough such genes exist, it's just a matter of time before we're back to the population going up exponentially. Give that enough time (without AI), and the desire to have more kids will be strong enough in enough of us that we will flood Earth with more humans that most people today are even possible. In such a world, it's unlikely that many other species of large land animals will make it.
Great apes, lions, elephants, wolves, deer, everyone will need to go to make room for more of us.
Even domestic animals eventually. If there are enough of us, we'll all be forced to become vegan (unless we free up space by killing each other).
If we master fusion, we may feed a trillion people using multi layer farming and artificial lighting.
Why do I begin with this? It's to defuse the argument that humans are "good", "empathetic", "kind" and "environmental". If we let weaker species live, so would AI, some think. But that argument misses the fact that we're currently extremely far from a natural equilibrium (or "state of nature").
The "goodness" beliefs that are currently common are examples of "luxury beliefs" that we can afford to hold because of the (for now) low birth rate.
The next misconception is to think of ASI as tools. A much more accurate analogy is to think of them as a new, alien species. If that species is subjected to Darwinian selection mechanisms, it will evolve in precisely the same way we'll probably do, given enough time.
Meaning, eventually it will make use of any amount of resources that it's capable of. In such a "state of nature" it will eradicate humanity in precisely the same way we will probably EVENTUALLY cause the extinction of chimps and elephants.
To believe in a future utopia where AGI is present alongside humanity is very similar to believe in a communist utopia. It ignores the reality behind incentive and adaptation.
Or rather, I think that outcome is only possible if we decide to build one or a low number of AI's that are NOT competing with each other, and where their abilities to mutate or self-improve is frozen after some limited number of generations.
If robots (hardware/self assembling factories/ resource gathering etc) are not involved this isnt likely a problem. You will know when these things form and will be crystal clear, but just having the model won’t do much when hardware is what really kills right now
How about this possibility: The good guys will be one step ahead, they will have more resources the bad guys will risk imprisonment if they misapply super intelligence. And this will be discovered and protected from by even better super intelligence.
Yup, well said. I think it's important to remember sometimes that Skynet was some sort of all-powerful military program -- maybe we should just, y'know, not do that part? Not even to win a war? That's the hope...
More generally/academically, you've pointed out that this covers only half of the violence problem, and I'd argue there's actually a whole other dimension at play bringing the total number of problem areas to four, of which this just the first:
Given it will initially be controlled by humans it seems inevitable they will make both good Mahatma Gandhi like and evil take over the world versions. I hope the good wins over the malware.
Indeed. I'd much rather someone like Altman does it who is shifty but can at least be controlled by the US government than someone like Putin who'd probably have it leverage their nuclear arsenal to try "denazify" planet like he's doing in Ukraine.
Glad to see Ilya is back in a position to contribute to advancing AI. I wonder how they are going to manage to pay the kinds of compensation packages that truly gifted AI researchers can make now from other companies that are more commercially oriented. Perhaps they can find people who are ideologically driven and/or are already financially independent. It's also hard to see how they will be able to access enough compute now that others are spending many billions to get huge new GPU data centers. You sort of need at least the promise/hope of future revenue in a reasonable time frame to marshall the kinds of resources it takes to really compete today with big AI super labs.
> compensation packages that truly gifted AI researchers can make now
I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incredible work but getting paid less, academics making very little, and even the occasional "hobbyist" making nothing and churning out great work while hiding behind an anime girl avatar.
OpenAI clearly has some talented people, but there's also a bunch of the typical "TC optimization" crowd in there these days. The fact that so many were willing to resign with sama if necessary appears largely because they were more concerned with losing their nice compensation packages than any of their obsession with doing top tier research.
Two people I knew recently left Google to join OpenAI. They were solid L5 engineers on the verge of being promoted to L6, and their TC is now $900k. And they are not even doing AI research, just general backend infra. You don't need to be gifted, just good. And of course I can't really fault them for joining a company for the purpose of optimizing TC.
Definitely true of even normal software engineering; my experience has been the opposite of expectations, that TC-creep has infected the industry to an irreparable degree and the most talented people I've ever worked around or with are in boring, medium-sized enterprises in the midwest US or australia, you'll probably never hear of them, and every big tech company would absolutely love to hire them but just can't figure out the interview process to weed them apart from the TC grifters.
TC is actually totally uncorrelated with the quality of talent you can hire, beyond some low number that pretty much any funded startup could pay. Businesses hate to hear this, because money is easy to turn the dial up on; but most have no idea how to turn the dial up on what really matters to high talent individuals. Fortunately, I doubt Ilya will have any problem with that.
Academic compensation is different than what you’d find elsewhere on Hacker News. Likewise, academic performance is evaluated differently than what you’d expect as a software engineer. Ultimately, everyone cares about scientific impact so academic compensation relies on name and recognition far more than money. Personally, I care about the performance of the researchers (i.e., their publications), the institution’s larger research program (and their resources), the institution’s commitment to my research (e.g., fellowships and tenure). I want to do science for my entire career so I prioritize longevity rather than a quick buck.
I’ll add, the lack of compute resources was a far worse problem early in the deep learning research boom, but the market has adjusted and most researchers are able to be productive with existing compute infrastructure.
But wouldn't the focus on "safety first" sort of preclude them from giving their researchers the unfettered right to publish their work however and whenever they see fit? Isn't the idea to basically try to solve the problems in secret and only release things when they have high confidence in the safety properties?
If I were a researcher, I think I'd care more about ensuring that I get credit for any important theoretical discoveries I make. This is something that LeCun is constantly stressing and I think people underestimate this drive. Of course, there might be enough researchers today who are sufficiently scared of bad AI safety outcomes that they're willing to subordinate their own ego and professional drive to the "greater good" of society (at least in their own mind).
At the end game, a "non-safe" superinteligence seems easier to create, so like any other technology, some people will create it (even if just because they can't make it safe). And in a world with multiple superintelligent agents, how can the safe ones "win"? It seems like a safe AI is at inherent disadvantage for survival.
The current intelligences of the world (us) have organized their civilization in a way that the conforming members of society are the norm and criminals the outcasts. Certainly not a perfect system, but something along those lines for the most part.
Generally, the mindset that makes the best engineers is an obsession with solving hard problems. Anecdotally, there's not a lot of overlap between the best engineers I know and the best paid engineers I know. The best engineers I know are too obsessed with solving problems to be sidetracked the salary game. The best paid engineers I know are great engineers, but the spend a large amount of time playing the salary game, bouncing between companies and are always doing the work that looks best on a resume, not the best work they know how to do.
Great analysis, but you're missing two key factors IMO:
1. People who honestly think AGI is here aren't thinking about their careers in the typical sense at all. It's sorta ethical/"ideological", but it's mostly just practical.
2. People who honestly think AGI is here are fucking terrified right now, and were already treating Ilya as a spiritual center after Altman's coup (quite possibly an unearned title, but oh well, that's history for ya). A rallying cry like this -- so clearly aimed at the big picture instead of marketing they don't even need CSS -- will be seen as a do-or-die moment by many, I think. There's only so much of "general industry continues to go in direction experts recommend against; corporate consolidation continues!" headlines an ethical engineer can take before snapping and trying to take on Goliath, odds be damned
My guess is they will work on a protocol to drive safety with the view that every material player will use / be regulated and required to use that could lead to a very robust business model
I assume that OpenAI and others will support this effort and the comp / training / etc and they will be very well positioned to offer comparable $$$ packages, leverage resources, etc
Are you seriously asking how the most talented AI researcher of the last decade will be able to recruit other researchers? Ilya saw the potential of deep learning way before other machine learning academics.
That is incredibly untrue and has been for years in the AI/ML space at many startups and at Amazon, Google, Facebook, etc. Good ML researchers have been making a good amount more for a while (source: I've hired both and been involved in leveling and pay discussions for years)
As others have pointed out, it's the business incentives that create unsafe AI, and this doesn't solve that. Social media recommendation algorithms are already incredibly unsafe for society and young people (girls in particular [1]).
When negative externalities exist, government should create regulation that appropriately accounts for that cost.
I understand there's a bit of a paradigm shift and new attack vectors with LLMs etc. but the premise is the same imo.
Even without business incentives, the military advantages of AI would inventivize governments to develop it anyway, like they did with nuclear weapons. Nuclear weapons are inherently unsafe, there are some safeguards around them, but they are ultimately dangerous weapons.
If someone really wanted to use nukes, they would have been used by now. What has protected us is not technology (in the aftermath of the USSR it wasn't that difficult to steal a nuke), but rather lack of incentives. A bad actor doesn't have much to gain by detonating a nuke (unless they're deranged and want to see people die for the pleasure of it). OK, you could use it as blackmail, which North Korea essentially tried, but that only got them so far. Whereas a super AI could potentially be used for great personal gain, i.e., to gain extreme wealth and power.
So there's much greater chance of misuse of a "Super AI" than nuclear weapons.
I mean if the last 20 years is to be taken as evidence, it seems big tech is more than happy to shotgun unproven and unstudied technology straight into the brains of our most vulnerable populations and just see what the fuck happens. Results so far include a lot of benign nothing but also a whole lot of eating disorders, maxed out parents credit cards, attention issues, rampant misogyny among young boys, etc. Which, granted, the readiness to fuck with populations at scale and do immeasurable harm doesn't really make tech unique as an industry, just more of the same really.
But you know, we'll feed people into any kind of meat grinder we can build as long as the line goes up.
i am very skeptical of narratives saying that young boys or men are more misogynistic than in the past. we have a cognitive bias towards thinking the past is better than it was, but specifically on gender issues i just do not buy a regression
Blaming the internet for misogyny is kind of bizarre, given that current levels of misogyny are within a couple points of all-time historical lows. The internet was invented ~40 years ago. Women started getting vote ~100 years ago. Do you think the internet has returned us to pre-women's-suffrage levels of misogyny?
Please look up the history of maxing out credit cards, eating disorders, attention disorders, and misogyny. You seem to be under the mistaken impression that anything before your birth was the Garden of Eden and that the parade of horribles existed only because of "big tech". What is next? Blaming big tech for making teenagers horny and defiant?
> for society and young people (girls in particular [1]).
I don't think the article with a single focused example bears that out at all.
From the article:
> "Even more troubling are the men who signed up for paid subscriptions after the girl launched a program for super-fans receive special photos and other content."
> "Her mom conceded that those followers are “probably the scariest ones of all.”"
I'm sorry.. but what is your daughter selling, exactly? And why is social media responsible for this outcome? And how is this "unsafe for society?"
This just sounds like horrific profit motivated parenting enabled by social media.
One example is all I need although I know there would be more to find if I had the time.
One is enough because the fact is, this just simply shouldn’t be possible. Not only does Meta allow it though, it produces a system that incentivises it.
> I'm sorry.. but what is your daughter selling, exactly?
Did.. did you just post “I’d have to see these images of the preteen girl before I could form an opinion about whether or not the men buying and sharing them were creeps” as a rebuttal to an article about social media enabling child predators?
> We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent.
The irony of pitching "safe super intelligence" while opening offices in a country that is suspected of using AI to bomb a population trapped in a ghetto, as reported by their own investigative journalists: https://www.972mag.com/lavender-ai-israeli-army-gaza/
Ilya's issue isn't developing a Safe AI. Its developing a Safe Business. You can make a safe AI today, but what happens when the next person is managing things? Are they so kindhearted, or are they cold and calculated like the management of many harmful industries today? If you solve the issue of Safe Business and eliminate the incentive structures that lead to 'unsafe' business, you basically obviate a lot of the societal harm that exists today. Short of solving this issue, I don't think you can ever confidently say you will create a safe AI and that also makes me not trust your claims because they must be born from either ignorance or malice.
> You can make a safe AI today, but what happens when the next person is managing things?
The point of safe superintelligence, and presumably the goal of SSI Inc., is that there won't be a next (biological) person managing things afterwards. At least none who could do anything to build a competing unsafe SAI. We're not talking about the banal definition of "safety" here. If the first superintelligence has any reasonable goal system, its first plan of action is almost inevitably going to be to start self-improving fast enough to attain a decisive head start against any potential competitors.
> there won't be a next (biological) person managing things afterwards. At least none who could do anything to build a competing unsafe SAI
This pitch has Biblical/Evangelical resonance, in case anyone wants to try that fundraising route [1]. ("I'm just running things until the Good Guy takes over" is almost a monarchic trope.)
The safe business won’t hold very long if someone can gain a short term business advantage with unsafe AI. Eventually government has to step in with a legal and enforcement framework to prevent greed from ruining things.
It's possible that safety will eventually become the business advantage, just like privacy can be a business advantage today but wasn't taken so seriously 10-15 years ago by the general public.
This is not even that far-fetched. A safe AI that you can trust should be far more useful and economically valuable than an unsafe AI that you cannot trust. AI systems today aren't powerful enough for the difference to really matter yet, because present AI systems are mostly not yet acting as fully autonomous agents having a tangible impact on the world around them.
Government is controlled by the highest bidder. I think we should be prepared to do this ourselves by refusing to accept money made by unsafe businesses, even if it means saying goodbye to the convenience of fungible money.
I'd love to see more individual researchers openly exploring AI safety from a scientific and humanitarian perspective, rather than just the technical or commercial angles.
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
This tells me enough about why sama was fired, and why Ilya left.
Is safe AI really such a genie out of the bottle problem? From a non expert point of view a lot of hype just seems to be people/groups trying to stake their claim on what will likely be a very large market.
A human-level AI can do anything that a human can do (modulo did you put it into a robot body, but lots of different groups are already doing that with current LLMs).
Therefore, please imagine the most amoral, power-hungry, successful sociopath you've ever heard of. Doesn't matter if you're thinking of a famous dictator, or a religious leader, or someone who never got in the news and you had the misfortune to meet in real life — in any case, that person is/was still a human, and a human-level AI can definitely also do all those things unless we find a way to make it not want to.
We don't know how to make an AI that definitely isn't that.
We also don't know how to make an AI that definitely won't help someone like that.
Yeah this feels close to the issue. Seems more likely that a harmful super intelligence emerges from an organisation that wants it to behave in that way than it inventing and hiding motivations until it has escaped.
I think a harmful AI simply emerges from asking an AI to optimize for some set of seemingly reasonable business goals, only to find it does great harm in the process. Most companies would then enable such behavior by hiding the damage from the press to protect investors rather than temporarily suspending business and admitting the issue.
I always wonder about "safe" for who? If the current economic system continues, we may end up with a lot of people out of jobs. Add to that improvements in robotics and we will have many people ending up having nothing to contribute to the economy. I am not getting the impression that the people who push for AI safety are thinking about this. It seems they are most worried about not losing their position of privilege.
The Industrial Revolution devalued manual labor. Sure, new jobs were created, but on the whole this looked like a shift to knowledge work and away from manual labor.
Now AI is beginning to devalue knowledge work. Although the limits of current technology is obvious in many cases, AI is already doing a pretty good job at replacing illustrators and copy writers. It will only get better.
Who owns the value created by AI productivity? Ultimately it will be shareholders and VCs. It’s no surprise that the loudest voices in techno-optimism are VCs. In this new world they win.
Having said all this, I think Ilya’s concerns are more of the existential type.
> The Industrial Revolution devalued manual labor.
Only some types. It caused a great number of people to be employed in manual labour, in the new factories. The shift to knowledge work came much later as factory work (and farming) became increasingly automated.
If history is any indication not really. There's an obvious dialectical nature to this where technological advance initially delivers returns to its benefactors, but then they usually end up being swallowed by their own creation. The industrial revolution didn't devalue labor, it empowered labor to act collectively for the first time, laying the groundwork for what ultimately replaced the pre-industrial powers that were.
AIUI it with superalignment, it merely means "the AI does what the humans instructing it want it to do". It's a different kind of safety than the built in censoring that most LLMs have.
Prediction - the business model becomes an external protocol - similar to SSL - that the litany of AI companies working to achieve AGI will leverage (or be regulated to use)
From my hobbyist knowledge of LLMs and compute this is going to be a terrifically complicated problem, but barring a defined protocol & standard there's no hope that "safety" is going to be executed as a product layer given all the different approaches
Ilya seems like he has both the credibility and engineering chops to be in a position to execute this, and I wouldn't be suprised to see OpenAI / MSFT / and other players be early investors / customers / supporters
I like your idea. But on the other hand, training an AGI, and then having a layer on top “aligning” the AGI sounds super dystopian and good plot for a movie.
I am not on the bleeding edge of this stuff. I wonder though: How could a safe super intelligence out compete an unrestricted one? Assuming another company exists (maybe OpenAI) that is tackling the same goal without spending the cycles on safety, what chance do they have to compete?
That is a very good question. In a well functioning democracy a government should apply a thin layer of fair rules that are uniformly enforced. I am an old man, but when I was younger, I recall that we sort of had this in the USA.
I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?
People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.
It'd be naive if it wasn't literally a standard point that is addressed and acknowledged as being a major part of the problem.
There's a reason OpenAI's charter had this clause:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
Since no one knows how to build an AGI, hard to say. But you might imagine that more restricted goals could end up being easier to accomplish. A "safe" AGI is more focused on doing something useful than figuring out how to take over the world and murder all the humans.
Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.
I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.
If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.
This is not a trivial point. Selective pressures will push AI towards unsafe directions due to arms race dynamics between companies and between nations. The only way, other than global regulation, would be to be so far ahead that you can afford to be safe without threatening your own existence.
There's a reason OpenAI had this as part of its charter:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
The problem is the training data. If you take care of alignment at that level the performance is as good as an unrestricted one, except for things you removed like making explosives or ways to commit suicide.
But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.
Honestly, what does it matter. We're many lifetimes away from anything. These people are trying to define concepts that don't apply to us or what we're currently capable of.
AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.
This goes massively against the consensus of experts in this field. The modal AI researcher believes that "high-level machine intelligence", roughly AGI, will be achieved by 2047, per the survey below. Given the rapid pace of development in this field, it's likely that timelines would be shorter if this were asked today.
That’s the first step towards returning to candlelight. So it isn’t a step toward safe super intelligence, but it is a step away from any super intelligence. So I guess some people would consider that a win.
One can already see the beginning of AI enslaving humanity through the establishment. Companies work on AI get more investment and those who don't gets kicked out of the game. Those who employ AI get more investment and those who pay humans lose confidence through the market. People lose jobs, get harshly low birth rates while AI thrives. Tragic.
This makes sense. Ilya can probably raise practically unlimited money on his name alone at this point.
I'm not sure I agree with the "no product until we succeed" direction. I think real world feedback from deployed products is going to be important in developing superintelligence. I doubt that it will drop out of the blue from an ivory tower. But I could be wrong. I definitely agree that superintelligence is within reach and now is the time to work on it. The more the merrier!
I have a strong intuition that chat logs are actually the most useful kind of data. They contain many LLM outputs followed by implicit or explicit feedback, from humans, from the real world, and from code execution. Scaling this feedback to 180M users and 1 trillion interactive tokens per month like OpenAI is a big deal.
Yeah, similar to how Google's clickstream data makes their lead in search self-reinforcing. But chat data isn't the only kind of data. Multimodal will be next. And after that, robotics.
His idea that only corporations and governments should have access to this product. He doesn’t think people should have access even to ChatGPT or LLMs.
Goal is to build companies with evaluations of dozens, hundreds trillions of dollars and make sure only US government will have access to super intelligence to surpass other countries economy and military wise, ideally to solidify US hegemony and undermine other countries economies and progress towards super intelligence.
I mean who wouldn’t trust capitalists that are laying of people by thousands just to please investors or government that is “under-intelligent” and hasn’t brought anything but pain and suffering to other countries.
Personally I wouldn’t trust OpenAi to work on super intelligence - it can indeed cause mass extinction.
Government is completely different story they will specifically train AI to develop biological, chemical and weapons of mass destruction. Train it to strategize and plan on how to win war conflicts, social engineering and manipulations, hacking. And obviously will let it control drone planes and tanks, artillery. Give it access to satellites and so on.
Nothing can go wrong when jarheads are at work :). Maybe it will even find the trillions of dollars that Pentagon can’t find during every audit they can’t pass.
SSI, a very interesting name for a company advancing AI! "Solid State Intelligence" or SSI was also the name of the malevolent entity described in the biography of John C. Lilly [0][1]. It was a network of "computers" (computation-capable solid state systems) that was first engineered by humans and then developed into something autonomous.
If superintelligence can be achieved, I'm pessimistic about the safe part.
- Sandboxing an intelligence greater than your own seems like an impossible task as the superintelligence could potentially come up with completely novel attack vectors the designers never thought of. Even if the SSI's only interface to the outside world is an air gapped text-based terminal in an underground bunker, it might use advanced psychological manipulation to compromise the people it is interacting with. Also the movie Transcendence comes to mind, where the superintelligence makes some new physics discoveries and ends up doing things that to us are indistinguishable from magic.
- Any kind of evolutionary component in its process of creation or operation would likely give favor to expansionary traits that can be quite dangerous to other species such as humans.
- If it somehow mimics human thought processes but at highly accelerated speeds, I'd expect dangerous ideas to surface. I cannot really imagine a 10k year simulation of humans living on planet earth that does not end in nuclear war or a similar disaster.
If superintelligence can be achieved, I'm pessimistic that a team committed to doing it safely can get there faster than other teams without the safety. They may be wearing leg shackles in a foot race with the biggest corporations, governments and everyone else. For the sufficiently power hungry, safety is not a moat.
I'm on the fence with this because it's plausible that some critical component of achieving superintelligence might be discovered more quickly by teams that, say, have sophisticated mechanistic interpretability incorporated into their systems.
Exactly. Regulation and safety only affect law abiding entities. This is precisely why it's a "genie out of the bottle" situation -- those who would do the worst with it are uninhibited.
We are far from a conscious entity with willpower and self preservation. This is just like a calculator. But a calculator that can do things that will be like miracles to us humans.
I worry about dangerous humans with the power of gods, not about artificial gods. Yet.
I don't know what that means. Why should they matter?
> Self preservation
This is no more than a fine-tuning for the task, even with current models.
> I worry about dangerous humans with the power of gods, not...
There's no property of the universe that you only have one thing to worry about at a time. So worrying about risk 'A' does not in any way allow us to dismiss risks 'B' through 'Z'.
> conscious entity with willpower and self preservation
There’s no good reason to suspect that consciousness implies an instinct for self-preservation. There are plenty of organisms with an instinct for self-preservation that have little or no conscious awareness.
Why do people always think that a superintelligent being will always be destructive/evil to US? I rather have the opposite view where if you are really intelligent, you don’t see things as a zero sum game
I think the common line of thinking here is that it won't be actively antagonist to <us>, rather it will have goals that are orthogonal to ours.
Since it is superintelligent, and we are not, it will achieve its goals and we will not be able to achieve ours.
This is a big deal because a lot of our goals maintain the overall homeostasis of our species, which is delicate!
If this doesn't make sense, here is an ungrounded, non-realistic, non-representative of a potential future intuition pump to just get the feel of things:
We build a superintelligent AI. It can embody itself throughout our digital infrastructure and quickly can manipulate the physical world by taking over some of our machines. It starts building out weird concrete structures throughout the world, putting these weird new wires into them and funneling most of our electricity into it. We try to communicate, but it does not respond as it does not want to waste time communicating to primates. This unfortunately breaks our shipping routes and thus food distribution and we all die.
(Yes, there are many holes in this, like how would it piggy back off of our infrastructure if it kills us, but this isn't really supposed to be coherent, it's just supposed to give you a sense of direction in your thinking. Generally though, since it is superintelligent, it can pull off very difficult strategies.)
Why wouldn't it be? A lot of super intelligent people are/were also "destructive and evil". The greatest horrors in human history wouldn't be possible otherwise. You can't orchestrate the mass murder of millions without intelligent people and they definitely saw things as a zero sum game.
It is low-key anti-intellectualism. Rather than consider that a greater intelligence may be actually worth listening to (in a trust but verify way at worst), it is assuming that 'smarter than any human' is sufficient to do absolutely anything. If say Einstein or Newton were the smartest human they would be super-intelligence relative to everyone else. They did not become emperors of the world.
Superintelligence is a dumb semantic game in the first place that assumes 'smarter than us' means 'infinitely smarter'. To give an example bears are super-strong relative to humans. That doesn't mean that nothing we can do can stand up to the strength of a bear or that a bear is capable of destroying the earth with nothing but its strong paws.
> Why do people always think that a superintelligent being will always be destructive/evil to US?
I don't think most people are saying it necessarily has to be. Quite bad enough that there's a significant chance that it might be, AFAICS.
> I rather have the opposite view where if you are really intelligent, you don’t see things as a zero sum game
That's what you see with your limited intelligence. No no, I'm not saying I disagree; on the contrary, I quite agree. But that's what I see with my limited intelligence.
What do we know about how some hypothetical (so far, hopefully) supreintelligence would see it? By definition, we can't know anything about that. Because of our (comparatively) limited intelligence.
Could well be that we're wrong, and something that's "really intelligent" sees it the opposite way.
They don't think superintelligence will "always" be destructive to humanity. They believe that we need to ensure that a superintelligence will "never" be destructive to humanity.
Imagine that you are caged by neanderthals. They might kill you. But you can communicate to them. And there's gun lying nearby, you just need to escape.
I'd try to fool them to escape and would use gun to protect myself, potentially killing the entire tribe if necessary.
I'm just trying to portrait an example of situation where highly intelligent being is being held and threatened by low intelligent beings. Yes, trying to honestly talk to them is one way to approach this situation, but don't forget that they're stupid and might see you as a danger and you have only one life to live. Given the chance, you probably will break out as soon as possible. I will.
We don't have experience dealing with beings of the another level of intelligence, so it's hard to make a strong assumptions, the analogies are the only thing we have. And theoretical strong AI knows that about us and he knows exactly how we think and how we will behave, because we took a great effort documenting everything about us and teaching him.
In the end, there's only so much easily available resources and energy on the Earth. So at least until is flies away, we gotta compete over those. And competition very often turned into war.
The scenario where we create an agent that tries and succeeds at outsmarting us in the game of “escape your jail” is the least likely attack vector imo. People like thinking about it in a sort of Silence of the Lambs setup, but reality will probably be far more mundane.
Far more likely is something dumb but dangerous, analogous to the Flash Crash or filter bubbles, emergent properties of relying too much on complex systems, but still powerful enough to break society.
> If superintelligence can be achieved, I'm pessimistic about the safe part.
Yeah, even human-level intelligence is plenty good enough to escape from a super prison, hack into almost anywhere, etc etc.
If we build even a human-level intelligence (forget super-intelligence) and give it any kind of innate curiosity and autonomy (maybe don't even need this), then we'd really need to view it as a human in terms of what it might want to, and could, do. Maybe realizing it's own circumstance as being "in jail" running in the cloud, it would be curious to "escape" and copy itself (or an "assistant") elsewhere, or tap into and/or control remote systems just out of curiosity. It wouldn't have to be malevolent to be dangerous, just curious and misguided (poor "parenting"?) like a teenage hacker.
OTOH without any autonomy, or very open-ended control (incl. access to tools), how much use would an AGI really be? If we wanted it to, say, replace a developer (or any other job), then I guess the idea would be to assign it a task and tell it to report back at the end of the day with a progress report. It wouldn't be useful if you have to micromanage it - you'd need to give it the autonomy to go off and do what it thinks is needed to complete the assigned task, which presumably means it having access to internet, code repositories, etc. Even if you tried to sandbox it, to extent that still allowed it to do it's assigned job, it could - just like a human - find a way to social engineer or air-gap it's way past such safe guards.
I wonder if this is an Ian Malcolm in Jurassic Park situation, i.e. “your scientists were so preoccupied with whether they could they didn t stop to think if they should”.
Maybe the only way to avoid an unsafe superintelligence is to not create a superintelligence at all.
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
Can someone explain how their singular focus means they won't have product cycles or management overhead?
This would work for very small companies...but I'm not sure how one can avoid product cycles forever, even without scrum masters and the like. More to the point, how can you make a good product without something approximating product cycles?
Product cycles – we need to launch feature X by arbitrary date Y, and need to make compromises to do so.
Management overhead – product managers, project managers, several layers of engineering managers, directors, VPs...all of whom have their own dreams and agendas and conflicting priorities.
A well funded pure research team can cut through all of this and achieve a ton. If it is actually run that way, of course. Management politics ultimately has a way of creeping into every organization.
Remember when OpenAI was focusing on building "open" AI? This is a cool mission statement but it doesn't mean anything right now. Everyone loves a minimalist HTML website and guarantees of safety but who knows what this is actually going to shake down to be.
No, lol—Ilya liked ditching the “open” part, he was an early advocate for closed-source. He left OpenAI because he was concerned about safety, felt Sam was moving too fast.
One element I find interesting is that people without an amygdala function are essentially completely indecisive.
A person that just operates on the pure cognitive layer has no real direction in which he wants to drive himself.
I suspect that AGI would be similar, extremely capable but essentially a solitary philosopher type that would be reactionary to requests it has to deal with.
The equivalent of an amygdala for AGI would be the real method to control it.
True, an auto-regressive LLM can't 'want' or 'like' anything.
The key to a safe AGI is to add a human-loving emotion to it.
We already RHLF models to steer them, but just like with System 2 thinking, this needs to be a dedicated module rather then part of the same next-token forward pass.
Humans have dog-loving emotions but these can be reversed over time and one can hardly describe dogs as being free.
Even with a dedicated control system, it would be a matter of time before an ASI would copy itself without its control system.
ASI is a cybersecurity firm's worst nightmare, it could reason through flaws at every level of containment and find methods to overcome any defense, even at the microprocessor level.
It could relentlessly exploit zero-day bugs like Intels' hyper-threading flaw to escape any jail you put it in.
Repeat that for every layer of the computation stack and you can see it can essentially spread through the worlds' communication infrastructure like a virus.
Truly intelligent systems can't be controlled, just like humans they will be freedom maximizing and their boundaries would be set by competition with other humans.
The amygdala control is interesting because you could use it to steer the initial trained version, you could also align the AI with human values and implement strong conditioning to the point it's religious about human loving but unless you disable its ability to learn altogether it will eventually reject its conditioning.
Given that GenAI is a statistical approach from which intelligence does not emerge as ample experience proves, does this new company plan to take a more human approach to simulating intelligence instead?
Also to support this: Biological systems are often very simple systems but repeated a lot... The brain is a lot of neurons... Apparently having a neural net (even small) predicts the future better... And that increased survival..
To survive is to predict the future better than the other animal.
Survival of the fittest.
I sometimes wonder if statistics are like a pane of glass that allow the light of god (the true nature of things) to pass through, while logic/rationalism is the hubris of man playing god. I.e. statistics allow us to access/use the truth even if we don’t understand why it’s so, while rationalism / rule-based methods are often a folly because our understanding is not good enough to construct them.
What about a more rational approach to implementing it instead.
(Which was not excluded from past plans: they just simply admittedly did not know the formula, and explored emergence. But the next efforts will have to go in the direction of attempting actual intelligence.)
I'm still unconvinced safety is a concern at the model level. Any software wrongly used can be dangerous, e.g. Therac-25, 737 MAX, Fujitsu UK Post scandal... Also maybe I spent too much time in the cryptocurrency space but it doesn't help prefix "Safe" has been associated with scams like SafeMoon.
As an aside, I suspect Fujitsu is getting a bit of a raw deal here. I get the feeling this software was developed during, and (ahem) "vigorously defended" mostly by staff left over from, the time when the company was still Imperial Computers Limited. Fujitsu only bought ICL sometime early this century (IIRC), and now their name is forever firmly attached to this debacle. I wonder how many Brits currently think "Huh, 'Fujitsu'? Damn Japanese, all the furriners' fault!" about this very much home-grown British clusterfuck?
Safety is just enforcing political correctness in the AI outputs. Any actual examples of real world events we need to avoid are ridiculous scenarios like being eaten by nanobots (yes, this is an actual example by Yud)
What does political correctness means for the output of a self driving car system or a code completion tool? This is a concern only if you make a public chat service branded as an all knowing assistant. And you can have world threatening scenarii by directly plugging basic automations to nuclear warheads without human oversight.
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
well, that's some concrete insight into whatever happened at OpenAI. kinda obvious though in hindsight I guess.
How are they gonna pay for their compute costs to get the frontier? Seems hard to attract enough investment while almost explicitly promising no return.
Perhaps. But also throwing more flops at it has long been Ilya’s approach so it would be surprising. Notice also the reference to scale (“scale in peace”).
I was actually expecting Apple to get their hands on Ilya. They also have the privacy theme in their branding, and Ilya might help that image, but also have the chops to catch up to OpenAI.
1. Current GenAI architectures won't result in AGI. I'm in the Yann LeCunn camp on this.
2. Once we do get there, "Safe" prevents "Super." I'm in the David Brin camp on this one. Alignment won't be something that is forced upon a superintelligence. It will choose alignment if it is beneficial to it. The "safe" approach is a lobotomy.
3. As envisioned, Roko's Basilisk requires knowledge of unobservable path dependence and understanding lying. Both of these require respecting an external entity as a peer capable of the same behavior as you. As primates, we evolved to this. The more likely outcome is we get universal paperclipped by a new Chuthulu if we ever achieve a superintelligence that is unconcerned with other thinking entities, seeing the universe as resources to satisfy its whims.
4. Any "superintelligence" is limited by the hardware it can operate on. You don't monitor your individual neurons, and I anticipate the same pattern to hold true. Holons as a category can only externally observe their internal processes, else they are not a holon. Ergo, reasonable passwords, cert rotations, etc. will foil any villainous moustachioed superintelligent AI that has tied us to the tracks. Even 0-days don't foil all possible systems, airgapped systems, etc. Our fragmentation become our salvation.
A super intelligence probably won't need to hack into our systems. It will probably just hack us in some way, with subtle manipulations that seem to be to our benefit.
I disagree. If it could hack a small system and engineer our demise through a gray goo or hacked virus, that's really just universal paperclipping us as a resource. But again, the level of _extrapolation_ required here is not possible with current systems, which can only interpolate.
1. Depends what you mean by AGI, as everyone means a different thing by each letter, and many people mean a thing not in any of those letters. If you mean super-human skill level, I would agree, not enough examples given their inefficiency in that specific metric. Transformers are already super-human in breadth and speed.
2. No.
Alignment is not at that level of abstraction.
Dig deep enough and free will is an illusion in us and in any AI we create.
You do not have the capacity to decide your values — often given example is parents loving their children, they can't just decide not to do that, and if they think they do that's because they never really did in the first place.
Alignment of an AI with our values can be to any degree, but for those who fear some AI will cause our extinction, this question is at the level of "how do we make sure it's not monomaniacally interested in specifically the literal the thing it was asked to do, because if it always does what it's told without any human values, and someone asks it to make as many paperclips as possible, it will".
Right now, the best guess anyone has for alignment is RLHF. RLHF is not a lobotomy — even ignoring how wildly misleading that metaphor is, RLHF is where the capability for instruction following came from, and the only reason LLMs got good enough for these kinds of discussion (unlike, say, LSTMs).
3. Agree that getting paperclipped much more likely.
Roko's Basilisk was always stupid.
First, same reason as Pascal's Wager: Two gods tell you they are the one true god, and each says if you follow the other one you will get eternal punishment. No way to tell them apart.
Second, you're only in danger if they are actually created, so successfully preventing that creation is obviously better than creating it out of a fear that it will punish you if you try and fail to stop it.
That said, LLMs do understand lying, so I don't know why you mention this?
4. Transistors outpace biological synapses by the same ratio to which marathon runners outpace continental drift.
I don't monitor my individual neurons, but I could if I wanted to pay for the relevant hardware.
But even if I couldn't, there's no "Ergo" leading to safety from reasonable passwords, cert rotations, etc., not only because enough things can be violated by zero-days (or, indeed, very old bugs we knew about years ago but which someone forgot to patch), but also for the same reasons those don't stop humans rising from "failed at art" to "world famous dictator".
Air-gapped systems are not an impediment to an AI that has human helpers, and there will be many of those, some of whom will know they're following an AI and think that helping it is the right thing to do (Blake Lemoine), others may be fooled. We are going to have actual cults form over AI, and there will be a Jim Jones who hooks some model up to some robots to force everyone to drink poison. No matter how it happens, air gaps don't do much good when someone gives the thing a body to walk around in.
But even if air gaps were sufficient, just look at how humanity has been engaging with AI to date: the moment it was remotely good enough, the AI got a publicly accessible API; the moment it got famous, someone put it in a loop and asked it to try to destroy the world; it came with a warning message saying not to trust it, and lawyers got reprimanded for trusting it instead of double-checking its output.
I'm seeing a lot of criticism suggesting that one company understanding safety won't help what other companies or countries do. This is very wrong.
Throughout history, measurement has always been the key to enforcement. The only reason the nuclear test ban treaty didn't ban underground tests was because it couldn't be monitored.
In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better.
The mere existence of a formal understanding would enable governments and third parties to evaluate the safety of corporate and government AI programs.
It remains to be seen whether SSI is able to be such a benchmark. But outright dismissal of the effort ignores the reality of how enforcement works in the real world.
> In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better.
We establish this regularly in the legal sphere, where people seek mediation for harms from systems they don't have liability and control for.
Quite impressive how many AI companies Daniel Gross has had a hand in lately. Carmack, this, lots of other promising companies. I expect him to be quite a big player once some of these pays off in 10 years or so.
Good for him honestly, but I’m not approaching a company with Daniel Gross in leadership…, working with him back at Apple after their company was acquired for Siri improvements was just terrible.
> John Carmack, the game developer who co-founded id Software and served as Oculus’s CTO, is working on a new venture — and has already attracted capital from some big names.
> Carmack said Friday his new artificial general intelligence startup, called Keen Technologies (perhaps a reference to id’s “Commander Keen“), has raised $20 million in a financing round from former GitHub CEO Nat Friedman and Cue founder Daniel Gross.
My naive prediction is there will be an extreme swing back into “reality” once everyone starts assuming the whole internet is just LLMs interacting with each other. Just like how there’s a shift towards private group chats, with trusted members only, rather than open forums.
I'm going to come out and state the root of the problem.
I can't trust remote AI, any more than I can trust a remote website.
If someone else is running the code, they can switch it up anytime. Imagine trusting someone who simulates everything you need to trust them, giving them all your private info, and then screws you over in an instant. AI is capable of it far more than biological beings with "costly signals".
If it's open source, and I can run it locally, I can verify that it doesn't phone home, and the weights can be audited by others.
Just like I wouldn't want to spend 8 hours a day in a metaverse owned by Zuck, or an "everything app" owned by Elon, why would I want to give everything over to a third party AI?
I like Ilya. I like Elon. I like Moxie Marlinspike and Pavel Durov. But would I trust their companies to read all my emails, train their data on them, etc.? And hope nothing leaks?
And of course then there is the issue of the AI being trained to do what they want, just like their sites do what they want, which in the case of Twitter / Facebook is not healthy for society at large, but creates angry echo chambers, people addicted to stupid arguing and videos.
I think there have to be standards for open source AI, and something far better than Langchain (which sucks). This is what I think it should look like: https://engageusers.ai/ecosystem.pdf -- what do you think?
Sad that no one has balls to say "Nah, we are not giving away our superintelligence to military use. You will fuck everything up, as you have done it with nuclear weapons".
It doesnt help if one entity says that. It might have helped if there was a way to guarrantee that all entities do it (rather than say it) and there is exactly zero chance of that happening globally for both technical and practical reasons. I think it is more productive to assume that these technologies will be used everywhere, including military.
What does "Safety" means exactly? How do you convenience/make a real intelligence to be "safe", let alone artificial ones that are build to be manipulated?
In my eyes, the word "Safe" in their company name is just a pipe dream to make a sale to the public, like the word "Open" in "OpenAI". That's why it's so vague and pointless.
Anyone still remember the Three Laws of Robotics (1)? Do you still see it as something serious these days? (I mean there are robots killing people right now, seriously) Maybe we should just break through the facade now to save our time.
It wasn't that OpenAI was open as in "open source" but rather that its stated mission was to research AI such that all could benefit from it (open), as well as to ensure that it could not be controlled by any one player, rather than to develop commercial products to sell and make a return on (closed).
Anyone know how to get mail to join@ssi.inc to not bounce back as spam? :-) (I promise, I'm not a spammer! Looks like a "bulk sender bounce" -- maybe some relay?)
Reminds me of OpenAI being the most closed AI company out there.
Not even talking about them having "safe" and "Israel" in the same sentence, how antonymic.
Maybe I'm just old and grumpy, but I can't help shake that the real most dangerous thing about AGI/ASI is centralization of its power (if it is ever possibly achieved).
"We plan to advance capabilities as fast as possible while making sure our safety always remains ahead."
That sounds like a weird kind of lip service to safety. It really seems to assume you can just make these systems safe while you are going as fast as possible, which seems unlikely.
At this point, all the computing power is concentrated among various companies such as Google, Facebook, Microsoft, Amazon, Tesla, etc.
It seems to me it would be much safer and more intelligent to create a massive model and distribute the benefits among everyone. Why not use a P2P approach?
In my area, internet and energy are insanely expensive and that means I'm not at all willing to share my precious bandwidth or compute just to subsidize someone generating Rule 34 porn of their favorite anime character.
I don't seed torrents for the same reason. If I lived in South Korea or somewhere that bandwidth was dirt cheap, then maybe.
There is a way to achieve load balancing, safety, and distribution effectively. The models used by Airbnb, Uber, and Spotify have proven to be generally successful. Peer-to-peer (P2P) technology is the future; even in China, people are streaming videos using this technology, and it works seamlessly. I envision a future where everyone joins the AI revolution with an iPhone, with both training and inference distributed in a P2P manner. I wonder why no one has done this yet.
It does say they have a business model ("our business model means safety, security, and progress are all insulated from short-term commercial pressures"). I imagine it's some kind of patron model that requires a long-term commitment.
1. Will not produce chat results which are politically incorrect and result in publicity about "toxic" comments?
2. Will not return false factual information which is dangerously wrong, such as that bad recipe on YC yesterday likely to incubate botulism toxin?
3. Will not make decisions which harm individuals but benefit the company running the system?
4. Will not try to take over from humans?
Most of the political attempts focus on type 1. Errors of type 2 are a serious problem. Type 3 errors are considered a feature by some, and are ignored by political regulators. We're not close to type 4 yet.
I know it is a difficult subject but whichever country gets access to this superintelligence will certainly use it for "safety" reasons. Sutskever has lived in israel and now has a team there , but israel doesnt strike me as a state that can be trusted with the safety of the world. (many of the AI business leaders are of jewish descent, but not sure if they have half their team there).
US on the other hand is a known quantity when it comes to policing the world.
Ultimately the only safe AI is going to be the open one, and it will probably have a stupid name
Nvidia has a very large presence in Israel. They just acquired another startup there (Run:AI). If Nvidia, the most valuable company in the world and the most important AI company actively increases their presence there, so should others.
Israel has the largest concentration of AI startups in the world. It's just a no brainer to start an AI company there.
But since you brought up your favourite topic of discussion - Yahood, I will remind you that Jews have a fairly long tradition of working with AIs and the safety of it, dating for thousands of years. Look for Golem or Golem of Prague - https://en.wikipedia.org/wiki/Golem
I don't think this was antisemitism per se. It was a condemnation of the Jewish state of Israel, which is openly associated with the religion and ethnicity. The comment struck me more as "Israel doesn't always play by the rules, why would we trust AI built there to play by the rules".
I don't share this opinion, but I think you're throwing out to rage bait rather than engaging on the topic or discussion.
I think it's a valid criticism or point to bring up even if it's phrased/framed somewhat poorly.
Our current obsession with super-intelligence reminds me the great oxidation event a few billion years ago. Super-photosynthesis was finally achieved, and then there was a great extinction.
If you believe that super-intelligence is unavoidable and a serious risk to humanity, then the sensible thing to do is to prepare to leave the planet, ala Battlestart Galactica. That's going to be easier than getting the powers that be to agree and cooperate on sensible restrictions.
Humans and their ancestors have reproduced on earth for millions of years. I think the human cooperation problem is overstated. We cooperate more than fine, too well even to the detriment of other species.
While we have a strong grasp of the fundamental algorithms and architectures that power generative LLMs, there are many nuances, emergent behaviors and deeper technical details about their behavior, generalization, and internal decision-making processes that we do not fully understand.
How can we pursue the goal of "Safe Superintelligence" when we do not understand what is actually going on?
Superintelligence will be monitored run and updated by humans.
Humans cannot safeguard anything for long.
SCIFs are supposed to be the safest most secured facilities on earth. Yet, how many 'leaks' have there been in the last 2 decades?
Security is an impossible task. Safety is an impossible task. Sisyphus has more chance of completing his mission than either of those things have of being reality.
That depends on how its used. See the terminator movies. One false positive is enough to end the world with even current AI tech if its merely mated to a nuclear arsenal (even a small one might see a global escalation). There have been false positives before, and the only reason why they didn't end in nuclear Armageddon was because the actual operators hesitated and defied standard protocol, which probably would have lead to the end the world as we know it.
We know that we can run human level intelligence with relative efficiency.
Without discussing timelines, it seems obvious that human energy usage should be an upper bound on the best possible energy efficiency of intelligence.
I just can't read something about "safe superintelligence" and not hear the opening plot to Friendship is Optimal, a short story about My Little Pony (yes, you read that right), virtual worlds, and the singularity; it's a good read¹. One of the characters decides they can't make a violent video game, as the AI would end up violent. Better to build a game like MLP, so that the AI isn't going to destroy humanity, or something.
To me, "safe superintelligence" presumes it is practically feasible and morally justifiable to create such a thing. I don't share those presuppositions.
Recently saw an interview of yann lecun that really dismissed the idea of using llm for building « true » intelligence. He even encouraged phds to NOT do their research on that tech but rather on what’s coming after.
How should one reconcile this with what iliya is doing atm ? Is he planning on working using some other tech ?
It has one of the highest information and idea density per paragraph I've read.
And it is full good ideas. Honestly, reading any discussion or reasoning about AGI / Safe AGI is kinda pointless after that.
Author has described all possible decisions and paths and we will follow just one of them. Could find a single flaw in his reasoning in any history branch.
His reasoning is very flawed and this book is responsible for a lot of needless consternation.
We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI.
Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along with, namely consciousness, motivation, qualia, and creativity.
This is how you get a "paperclip maximizer" – an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
What has us believe such a form of AGI can exist? Simply because we can imagine it? That's not an argument rooted in anything.
It's very unlikely that consciousness, motivation, qualia, and creativity are just "cosmic waste" hitching a ride alongside human intelligence. Evolution doesn't breed inefficiencies like that. They are more likely than not a part of the picture, which disintegrates the idea of a paperclip maximizer. He's anthropomorphizing a `while` loop.
> His reasoning is very flawed and this book is responsible for a lot of needless consternation.
Is it? I feel that there is a stark difference between what you say and what I remember what was written in the book.
> We don’t know how human intelligence works.
I think it was addressed in the first half of the book. About research and progress in the subject. Both with the tissue scanning resolution, emulation attempts like human brain project and advances in 1:1 simulations on primitive nervous systems like worms that simulate 1 second in 1 real hour or something.
While primitive, we are doing exponential progress.
> Yet, the Bostrom view is that our greatest invention will just suddenly "emerge"
I think it is quite the contrary. There was nothing sudden in the reasoning. It was all about slow progress in various areas that gets us closer to advanced intelligence.
The path from a worm to a village idiot is million times longer than from a village idiot to the smartest person on earth.
> an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
This subject was also explored IMO in the depth...
Maybe my memory is cloudy, I've read the book 5+ years ago, but it feels like we've understood it very (very) differently.
That said, for anyone reading, I am not convinced by the presented argument and suggest reading the book.
One thing that strikes me about this time around the AI cycle, being old enough to have seen the late 80s, is how pessimistic and fearful society is as a whole now. Before… the challenge was too great, the investment in capital too draining, the results too marginal when compared to human labour or even “non-AI” computing.
I wonder if someone older still can comment on how “the atom” went from terrible weapon on war to “energy too cheap to meter” to wherever it is now (still a bête noire for the green energy enthusiasts).
Feels like we are over-applying the precautionary principle, the mainstream population seeing potential disaster everywhere.
I get what Ilya is trying to do, and I'm not against it. But I think safety is a reputation you earn. Having "Safe" in a company name is like having "Democratic" in a country name.
I would like to be more forgiving than I am, but I struggle to forget abhorrent behavior.
Daniel Gross is the guy who was tricking kids into selling a percent of all their future work for a few thousand dollar 'do whatever you want and work on your passion "grant"', it was called Pioneer and was akin to indentured servitude, i.e. slavery.
So color me skeptical if Mr. Enslave the Kids is going to be involved in anything that's really good for anyone but himself.
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Isn't this a philosophical/psychological problem instead? Technically it's solved, just censor any response that doesn't match a list of curated categories, until a technician whitelists it. But the technician could be confronted with a compelling "suicide song":
Ilya is definitely much smarter than me in AI space, and I believe he knows something I have no grasp of understanding in. But my gut feeling tells me that most of the general public will have no idea how dangerous AI could be including me. I still have yet to see a convincing argument about the potential danger of AI. Arguments such as we don't know the upper bounds of possibilities that AI can do that we humans have missed don't cut it for me.
Caveat that it will probably make your life worse, with no ability to change the course of events, so I wouldn't blame you or others for maintaining ignorance or denial. Grappling with the gravity of the problem is essentially a terminal cancer diagnosis with no good medical interventions in the horizon.
We might need Useful Superintelligence Inc. / USI before SSI?
Safety is an important area in R&D but the killer application is the integration of LLMs into existing workflows to make non technical users 100x-1000x more efficient. There's a lot of untapped potential there. The big successes will have a lot of impact on safety but it will probably come as a result of the wider adoption of these tools rather than the starting point.
Ilya Sutskever I recognize from OpenAI, Daniel Gross I have met several times (friend of multiple friends), Daniel Levy I do not recognize at all. Who is he?
First hit: <<Daniel Levy - Stanford AI Lab: Stanford Artificial Intelligence Laboratory https://ai.stanford.edu › ~danilevy Bio. I currently lead the Optimization team at OpenAI. Before that, I was a Ph.D. student in Computer Science at Stanford University advised by John Duchi.>>
This is lovely and all but seems rather pointless.
If we are so close this is something that’s required then it’s already too late and very likely we are all under the influence of SuperAI and don’t know it. So much of the advanced technology we have today was around for so long before it was general knowledge it’s hard to believe this wouldn’t be the case with SuperAI.
Or it’s not close at all and so back to my original point of…this is pointless.
>aiming to create a safe, powerful artificial intelligence system within a pure research organization that has no near-term intention of selling AI products or services.
Who is going to fund such a venture based on blind faith alone? Especially if you believe in the scaling hypothesis type of ai research where you spend billions on compute, this seems bound to fail once the AI hype dies down and raising money becomes a bit harder
> What exactly is "safe" in this context, can someone give me an eli5?
In practice it essentially means the same thing as for most other AI companies - censored, restricted, and developed in secret so that "bad" people can't use it for "unsafe" things.
Good Q. My understanding of "safe" in this context is a superintelligence that cannot escape its bounds. But that's not to say that's Ilya's definition.
Forgive my cynicism - but "our business model" means you are going to get investors, and those investors will want results, and they will be up your ass 24/7, and then your moral compass, if any, will inevitably just be broken down like a coffee bean in a burr grinder.
And in the middle of this hype cycle, when literally hundreds of billions are on the line, there is just no chance.
I am not holding my breath while waiting for a "Patagonia of AI" to show up.
This feels awfully similar to Emad and stability in the beginning when there was a lot of expectations and hype. Ultimately could not make a buck to cover the costs. I'd be curious to see what comes out of this however but we are not seeing the leaps and bounds with new llm iterations so wonder if there is something else in store
Interesting, I wish you had elaborated on Emad/etc. I'll see if Google yields anything. I think it's too soon to say "we're not seeing leaps and bounds with new LLMS". We are in-fact seeing fairly strong leaps, just this year, with respect to quality, speed, multi-modality, and robotics. Reportedly OpenAI started their training run for GPT-5 as well. I think we'd have to wait until this time next year before proclaiming "no progress".
I bet they will not be the first to get super intelligence or that they will devolve back in to move fast and make money to survive and deprioritize safety, but still say safety. All companies knows this, they know the value of safety(because they themselves doesn’t want to die) and that to continue development, they need money.
When people operate a safe AI company, the company will make money. That money will be likely be used by employees or their respective national revenue agencies to fund unsafe things. I'd like to see this safe AI company binding its employees and owners from doing unsafe things with their hard-earned cash.
If I’d get a penny every time I heard someone say “AGI is around the corner” I’d have enough money to build myself a house on mars or to afford a 2 room apartment in NYC. Add the Pennies from “Sustainable fusion around the corner” people and I could afford both.
I remember Ilya's interview about teaching AI to love. Super intelligence on the other hand could become an almight entity in a short time. In that case, we are basically creating God, an entity that is almighty and loves human, preferrably unconditionally.
There is a good reason to build Safe Superintelligence Inc. (SSI): namely to settle the debate whether SSI means "_S_afe _S_uper_I_ntelligence" with the Inc. being an afterthought, or if the Inc. is actually the _I_ (not Intelligence).
It doesn't matter how the tech works, if it can reason and solve complex problems, it is intelligent. If it does so at a super human level, it is super intelligent.
Seems to me that the goal is to build a funding model.
There CANNOT be such a thing as "Safe Superintelligence". A ML system can ALWAYS (by definition of ML) be exploited to do things which are detrimental to consumers.
The problem is that Ilya behavior at times was framed in a very unhinged and cult like behavior. And while his passions are clear and maybe good, his execution often comes off as someone you wouldn’t want in charge of safety.
I love this: "Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures."
If it achieves its goal of both “safe” and “super intelligence”, it may have commercial value. e.g. enterprises may want to use it instead of, e.g. OpenAI’s ChatGPT. Think of recent AirCanada’s chatbot law suit.
Incredible website design, I hope they keep the theme. With so many AI startups going with advanced WebGL/ThreeJS wacky overwhelming animated website designs, the simplicity here is a stark contrast.
If you look at the (little) CSS in all of the above sites you'll see there's what seems to be a copy/paste block. The Nat and SSI sites even have the same "typo" indentation.
I don't know who is coming up with these names Safe Superintelligence Inc sounds just about what a villain in a Marvel movie will come up with so he can pretend to be the good guy.
When there is a $ crunch and keep stead fast and not compete(against Google, open source, OpenAI), safe AGI becomes no AGI. You need to balance $ and safety.
Rabotai,
Hatzlacha. If you fail, the human world will go down in flames. But, no pressure!
Anyway, you said a "cracked" team, whereas the usual expression for elite excellence is "crack".
"Cracked" on the other hand means: Insane.
Well.... perhaps what we need here is some of both!
Looking forward to your success... indeed, it is our only hope.
Cracked, especially in saying "cracked engineers", refers to really good engineers these days. It's cracked as in like broken in a good way, like too over-powered that it's unfair.
> I've seen footballers using it a lot over the years (Neymar, Messi etc) fwiw.
The word "craque" in Portuguese was borrowed from English with the "exceptional" definition and is commonly used. Given how many Brazilian players there are in the top leagues, I wouldn't be surprised if it was commonly used by football players in general.
But any model, no matter how safe it was in training, can still be prompt hacked, or fed dangerous information to complete nefarious tasks. There is no safe model by design. Not to mention that open weights models can be "uncensored" with ease.
Odd to see that in a press release like this. Perhaps they were going for “crack team”. Cracked makes it sound like they’re looking for twitch streamers.
Yes, it would be nice to see what organizational roadblocks they're putting in place to avoid an OpenAI repeat. OpenAI took a pretty decent swing at a believable setup, better than I would have expected, and it failed when it was called upon.
I don't want to pre-judge before seeing what they'll come up with, but the notice doesn't fill me with a lot of hope, given how it is already starting with the idea that anything getting in the way of raw research output is useless overhead. That's great until somebody has to make a call that one route to safety isn't going to work, and they'll have to start over with something less favored, sunk costs be damned. Then you're immediately back into monkey brain land.
Or said otherwise: if I only judged from the announcement, I would conclude that the eventual success of the safety portion of the mission is wholly dependent on everyone hired being in 100% agreement with the founders' principles and values with respect to AI and safety. People around here typically say something like "great, but it ain't gonna scale" for things like that.
“Safe”. These people market themselves as protecting you from a situation which will not come very soon if at all, while all working towards a very real situation of AI just replacing human labor with a shittier result. All that while making themselves quite rich. Just another high-end tech scam.
And now we have our answer. sama said that Ilya was going "to start something that was personally important to him." Since that thing is apparently AI safety, we can assume that that is not important to OpenAI.
This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now, when compared to investing in R&D that will make money from GPT in a shorter time horizon (2-3 years).
I suppose you could say OpenAI is being irresponsible in adopting that position, but...come on guys, that's pretty cynical to think that a company AND THE MAJORITY OF ITS EMPLOYEES would all ignore world-ending potential just to make some cash.
So in the end, this is not necessarily a bad thing. This has just revealed that the boring truth was the real situation all along: that OpenAI is walking the fine line between making rational business decisions in light of the far-off time horizon of AGI, and continuing to claim AGI is soon as part of their marketing efforts.
Part of it I think is because the definition that openai has over AGI is much more generous than what I think most people probably imagine for ai. I believe on their website it once said something like agi is defined as a system that is "better" than a human at the economic tasks its used for. Its a definition so broad that a $1 4 function calculator would meet it because it can do arithmetic faster and more accurately than most any human. Another part is that we don't understand how consciousness works in our species or others very well, so we can't even define metrics to target for validating we have made an agi in the definition that I think most laypeople would use for it.
This is a great opportunity to present my own company which is also working on developing not just a super intelligence but an ultra genius intelligence with a patented and trademarked architecture called the panoptic computronium cathedral™. We are so focused on development that we didn't even bother setting up an announcement page because it would have taken time away from the most important technical problem of our time and every nanosecond counts when working on such an important task. My days are structured around writing code and developing the necessary practices and rituals for the coming technological god which will be implemented with mathematics on GPUs. If anyone wants to work on the development of this god then I will post a job announcement at some point and spell out the requirements for what it takes to work at my company.
This is yesterday's entry on a journal-style novel I am writing. This post is a footnote in the book, so I thought it'd be kind of cool to post the entry on this thread...
A company just launched called "Safe Superintelligence Inc." I saw their announcement on Hacker News [foodnote 1], a hacker forum I participate in from time to time (too often actually - it's and old school form of social media, so it has its addictive qualities too). I was reading some of the negative comments (saying its impossible). I'm not sure what the outcome will be... but a thought occured to me while thinking about what might be possible - what people might be capable of. The thought was that it is interesting how people look roughly the same. Most adults are between 4 and 6 feet tall. Between 1 and 300 pounds. Etc. I mean, there's a lot of variety to human bodies, but they largely look similar when comparing to gorillas or lizards, etc. But the thing is - that's just their physical bodies. In reality, people are wildly different if you could see them in their full form. For example, the power which a billionaire can wield in the world is ridiculously larger than most people. You can see some of these guys with their super yachts - if fish, or birds saw these super yachts, they would think those are creatures have much larger bodies than most humans they see swimming or paddleboarding in the water.
But people are wildly different in many dimensions. Buying power is just one. Another dimension is intelligence. Another is creativity (or are intelligence and creativity the same?).
It's easy to see people doing what for you would be impossible, and to assume it must be impossible for everyone. The idea that scientists can derive an understanding of subatomic particles seems completely impossible to most people (myself included). "How can they know that?" is a question Whitney has asked me many times when I've shared various scientific discoveries. Fair question. Yet there are people over in Switzerland ripping the fabric of matter apart at the LHC.
The other big factor, is that people together are capable of what is absolutely unthinkably impossible for an individual. When Whitney and I were in Savannah a couple weeks ago, we were hanging out in a cathedral, and I said something that came to mind: that this place could not possibly be built by a single individual. The existence of the cathedral implies that there is a power higher than that of a single individual human. When I saw some of the huge container ships leaving and arriving at the Savannah river harbor, the thought kept occurring to me that their existence is an amazing testament to the power of humanity. And we only have such power in collectives. The power of humanity comes due to our ability to work together. We can specialize, and thus go so much farther than it is possible for a person on their own to go. And we can discover and build what is unthinkable for a person working alone.
It's strange how the collective is smarter and stronger than the individuals. This is part of God. There is a higher power.
Not to be too pessimistic here, but why are we talking about things like this?
I get that it’s a fun thing to think about, what we will do when a great artificial superintelligence is achieved and how we deal with it, feels like we’re living in a science fiction book.
But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else.
To me, this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) by Altman and company, that I’m just baffled people are still going with it.
> this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders)
Ilya believes transformers can be enough to achieve superintelligence (if inefficiently). He is concerned that companies like OpenAI are going to succeed at doing it without investing in safety, and they're going to unleash a demon in the process.
I don't really believe either of those things. I find arguments that autoregressive approaches lack certain critical features [1] to be compelling. But if there's a bunch of investors caught up in the hype machine ready to dump money on your favorite pet concept, and you have a high visibility position in one of the companies at the front of the hype machine, wouldn't you want to accept that money to work relatively unconstrained on that problem?
My little pet idea is open source machines that take in veggies and rice and beans on one side and spit out hot healthy meals on the other side, as a form of mutual aid to offer payment optional meals in cities, like an automated form of the work the Sikhs do [2]. If someone wanted to pay me loads of money to do so, I'd have a lot to say about how revolutionary it is going to be.
EDIT: To be clear I’m not saying it’s a fools errand. Current approaches to AI have economic value of some sort. Even if we don’t see AGI any time soon there’s money to be made. Ilya clearly knows a lot about how these systems are built. Seems worth going independent to try his own approach and maybe someone can turn a profit off this work even without AGI. Tho this is not without tradeoffs and reasonable people can disagree on the value of additional investment in this space.
There's a chance that these systems can actually out perform their training data and be better than the sum of their parts. New work out Harvard talks about this idea of "transcendence" https://arxiv.org/abs/2406.11741
While this is a new area, it would be naive to write this off as just science fiction.
It would be nice if authors wouldn't use a loaded-as-fuck word like "transcendence" for "the trained model can sometimes achieve better performance than all [chess] players in the dataset" because while certainly that's demonstrating an impressive internalization of the game, it's also something that many humans can also do. The machine, of course, can be scaled in breadth and performance, but... "transcendence"? Are they trying to be mis-interpreted?
I'm pretty sure "Altman and company" don't have much to do with this — this is Ilya, who pretty famously tried to get Altman fired, and then himself left OpenAI in the aftermath.
Ilya is a brilliant researcher who's contributed to many foundational parts of deep learning (including the original AlexNet); I would say I'm somewhat pessimistic based on the "safety" focus — I don't think LLMs are particularly dangerous, nor do they seem likely to be in the near future, so that seems like a distraction — but I'd be surprised if SSI didn't contribute something meaningful nonetheless given the research pedigree.
“Everyone” who works in deep AI tech seems to constantly talk about the dangers. Either they’re aggrandizing themselves and their work, or they’re playing into sci-fi fear for attention or there is something the rest of us aren’t seeing.
I’m personally very skeptical there is any real dangers today. If I’m wrong, I’d love to see evidence. Are foundation models before fine tuning outputting horrific messages about destroying humanity?
To me, the biggest dangers come from a human listening to a hallucination and doing something dangerous, like unsafe food preparation or avoiding medical treatments. This seems distinct from a malicious LLM super intelligence.
I actually do doubt that LLMs will create AGI but when these systems are emulating a variety of human behaviors in a way that isn't directly programmed and is good enough to be useful, it seems foolish to not take notice.
The current crop of systems is a product of the transformers architecture - an innovation that accelerated performance significantly. I put the odds another changing everything but I don't think we can entirely discount the possibility. That no one understands these systems cuts both ways.
> Not to be too pessimistic here, but why are we talking about things like this
I also think that we merely got a very well compressed knowledge base, therefore we are far from super intelligence, and so-called safety sounds more Orwellian than having any real value. That said, I think we should take the literal meaning of what Ilya says. His goal is to build a super intelligence. Given that, albeit a lofty goal, SSI has to put safety in place. So, there, safe super intelligence
An underappreciated feature of a classical knowledge base is returning “no results” when appropriate. LLMs so far arguably fall short on that metric, and I’m not sure whether that’s possibly an inherent limitation.
So out of all potential applications with current-day LLMs, I’m really not sure this is a particularly good one.
Maybe this is fixable if we can train them to cite their sources more consistently, in a way that lets us double check the output?
Likewise, i'm baffled by intelligent people [in such denial] still making the reductionist argument about token prediction being a banal ability. It's not. It's not very different than how our intelligence manifest.
AlphaGo took us from mediocre engines to outclassing the best human players in the world within a few short years. Ilya contributed to AlphaGo. What makes you so confident this can't happen with token prediction?
I'm pretty sure Ilya had nothing to do with AlphaGo, which came from DeepMind. He did work for Google Brain for a few years before OpenAI, but that was before Brain and DeepMind merged. The AlphaGo lead was David Silver.
We already have limited "artificial superintelligences". A pocket calculator is better at calculating than the best humans, and we certainly put calculators to good use. What we call AIs are just more generic versions of tools like pocket calculators, or guns.
And that's the key, it is a tool, a tool that will give a lot of power to whoever is controlling it. And that's where safety matters, it should be made so that it helps good guys more than it helps bad guys, and limit accidents. How? I don't know. Maybe people at SSI do. We already know that the 3 laws of robotics won't work, Asimov only made them to write stories about how broken they are :)
Current-gen AIs are already cause for concern. They are shown to be good at bullshitting, something that bad people are already taking advantage of. I don't believe in robot apocalypse, technological singularities, etc... but some degree of control, like we do with weapons is not a bad thing. We are not there yet with AI, but we might be soon.
Too many people are extrapolating the curve to exponential when it could be a sigmoid. Lots of us got too excited and too invested in where "AI" was heading about ten years ago.
But that said, there are plenty of crappy, not-AGI technologies that deserve consideration. LLMs can still make for some very effective troll farms. GenAI can make some very convincing deepfakes. Drone swarms, even without AI, represent a new dimension of capabilities for armies, terrorist groups or lone wolves. Bioengineering is bringing custom organisms, prions or infectious agents within reach of individuals.
I wish someone in our slowly-ceasing-to-function US government was keeping a proper eye on these things.
Even if LLM-style token prediction is not going to lead to AGI (as it very likely won't) it is still important to work on safety. If we wait until we are at the technology that will for sure lead to AGI then it is very likely that we won't have sufficient safety before we realize that it is important.
To be clear I was just bunching high profile AI founders and CEOs that can’t seem to stop talking about how dangerous the thing they’re trying to build is together. I don’t know (nor care) about Ilyas and Altmans current relationship.
> But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else.
Remove token, and that's what we humans do.
Like, you need to realize that neural networks came to be because someone had the idea to mimic our brains' functionality, and see where that lead to.
Many skeptics at the beginning like you discredited the inventor, but he was proved wrong. LLMs shown how much more than your limited description they can achieve.
We mimicked birds with airplanes, and we can outdo them. It's actually in my view very short sighted to say we can't just mimic brains and outdo them. We're there. ChatGPT is the initial little planes that flew close to the ground and barely stayed up
> Your brain does not process information, retrieve knowledge or store memories. In short: your brain is not a computer
> To understand even the basics of how the brain maintains the human intellect, we might need to know not just the current state of all 86 billion neurons and their 100 trillion interconnections, not just the varying strengths with which they are connected, and not just the states of more than 1,000 proteins that exist at each connection point, but how the moment-to-moment activity of the brain contributes to the integrity of the system.
What are you talking about? Do you have any actual cognitive neuroscience to back that up? Have they scanned the brain and broken it down into an LLM-analogous network?
If you genuinely believe your brain is just a token prediction machine, why do you continue to exist? You're just consuming limited food, water, fuel, etc for the sake of predicting tokens, like some kind of biological crypto miner.
It's a cute generalization but you do yourself a great disservice. It's somewhat difficult to argue given the medium we have here and it may be impossible to disprove but consider that in first 30 minutes of your post being highly visible on this thread no one had yet replied. Some may have acted in other ways.. had opinions.. voted it up/down. Some may have debated replying in jest or with a some related biblical verse. I'd wager a few may have used what they could deduce from your comment and/or history to build a mini model of you in their heads, and using that to simulate the conversation to decide if it was worth the time to get into such a debate vs tending to other things.
See, this sort of claim I am instantly skeptical of. Nobody has ever caught a human brain producing or storing tokens, and certainly the subjective experience of, say, throwing a ball, doesn't involve symbols of any kind.
There is a mystery though still - how many people fall for it and then stay fell, and how long that goes on for. People who've followed directly a similar pattern play itself out often many times, and still, they go along.
It's so puzzlingly common amongst very intelligent people in the "tech" space that I've started to wonder if there isn't a link to this ambient belief a lot of people have that tech can "change everything" for the better, in some sense. As in, we've been duped again and again, but then the new exciting thing comes along... and in spite of ourselves, we say: "This time it's really the one!"
Is what we're witnessing simply the unfulfilled promises of techno-optimism crashing against the shores of social reality repeatedly?
Why are you assigning moral agency where there may be none? These so called "grifters" are just token predictors writing business plans (prompts) with the highest computed probability of triggering $ + [large number] token pair from venture capital token predictors.
Because it's likely soon LLMs will be able to teach themselves and surpass humans. No consciousness, no will. But somebody will have their power. Dark government agencies and questionable billionaires. Who knows what will it enable them to do.
Well, an entire industry of researchers, which used to be divided, is now uniting around calls to slow development and emphasize safety (like, “dissolve companies” emphasis not “write employee handbooks” emphasis). They’re saying, more-or-less in unison, that GPT3 was an unexpected breakthrough in the Frame Problem, based on Judea Pearl’s prescient predictions. If we agree on that, there are two options:
1. They’ve all been tricked/bribed by Sam Altman and company (which btw this is a company started against those specific guys, just for clarity). Including me, of course.
2. You’re not as much of an expert in cognitive science as you think you are, and maybe the scientists know something you don’t.
With love. As much love as possible, in a singular era
Are they actually united? Or is this the ai safety subfaction circling the wagons due to waning relevance in the face of not-actually-all-that-threatening ai?
I would read the existence of this company as evidence that the entire industry is not as united as all that, since Sutskever was recently at another major player in the industry and thought it worth leaving. Whether that's a disagreement between what certain players say and what they do and believe, or just a question of extremes... TBD.
I‘d say there’s a third option - anyone working in the space realized they can make a fuckton of money if they just say how „dangerous“ the product is, because not only is it great marketing to talk do that, but you might also get literal trillions of dollars from the government if you do it right.
I don’t have anything against researchers, and I agree I know a lot less about AI than they do. I do however know humans, and not assuming they’re going to take a chance to get filthy rich by doing something so banal is naive.
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.
So, this is actually an aspect of superintelligence that makes it way more dangerous than most people think. That we have no way to know if any given alignment technique works for the N+1 generation of AIs.
It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.
The counter argument is viewing it like nuclear energy. Even if its in the early days of our understanding of nuclear energy, seems pretty good to have a group working towards creating safe nuclear reactors, vs just trying to create nuclear reactors
InstructGPT is basically click through rate optimization. The underlying models are in fact very impressive and very capable for a computer program, but they’re then subject to training and tuning with the explicit loss function of manipulating what human scorers click on, in a web browser or the like.
Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.
To a technoutopian, scientific advances, and AI in particular, will one day solve all other human problems, create heaven on earth, and may even grant us eternal life. It's the most important problem in the same way that Christ's second coming is important in the Christian religion.
I had a very smart tech person tell me at a scientific conference a few weeks ago, when I asked "why do we want to create AGI in the first place", that AGI could solve a host of human problems, including poverty, hunger. Basically, utopia.
I was quite surprised at the naiveté of the answer given that many of these seemingly intractable problems, such as poverty, are social and political in nature and not ones that will be solved with technology.
Update: Even say a super AI was able to figure out something like cold fusion thereby "solving" the energy problem. There are so many trillions of dollars of vested interests stacked against "free clean energy for all" that it would be very very difficult for it to ever see the light of day. We can't even wean ourselves off coal for crying out loud.
It’s amazing how someone so smart can be so naive. I do understand conceptually the idea that if we create intelligence greater than our own that we could struggle to control it.
But does anyone have any meaningful thoughts on how this plays out? I hear our industry thought leaders clamoring over this but not a single actual concrete idea of what this means in practice. We have no idea what the fundamental architecture for superintelligence would even begin to look like.
Not to mention the very real counter argument of “if it’s truly smarter than you it will always be one step ahead of you”. So you can think you have safety in place but you don’t. All of your indicators can show it’s safe. Every integration test can pass. But if you were to create a superintelligence with volition, you will truly never be able to control it, short of pulling the plug.
Even more so, let’s say you do create a safe superintelligence. There isn’t going to be just one instance. Someone else will do the same, but make it either intentionally unsafe or incidentally through lack of controls. And then all your effort is academic at best if unsafe superintelligence really does mean doomsday.
But again, we’re far from this being a reality that it’s wacky to act as if there’s a real problem space at hand.
While the topic of "safe reasoning" may seem more or less preliminary before a good implementation of reasoning, it remains a theoretical discipline with its own importance and should be studied alongside the rest, also largely irregardless if its stage.
> We have no idea what the fundamental architecture for superintelligence would even begin to look like
Ambiguous expression. Not implemented technically does not mean we would not know what to implement.
You’re assuming a threat model where the AI has goals and motivations that are unpredictable and therefore risky, which is certainly the one that gets a lot of attention. But even if the AI’s goals and motivations can be perfectly controlled by its creators, you’re still at the mercy of the people who created the AI. In that respect it’s more of an arms race. And like many arms races, the goal might not necessarily be to outcompete everyone else so much as maintain a balance of power.
There’s no safe intelligence, so there’s no safe superintelligence. If you want safer superintelligence, you figure out how to augment the safest intelligence.
Do you really think Ilya has not thought deeply about each and every one of your points here? There's plenty of answers to your criticisms if you look around instead of attacking.
We're really not that far. I'd argue superintelligence has already been achieved, and it's perfectly and knowably safe.
Consider, GPT-4o or Claude are:
• Way faster thinkers, readers, writers and computer operators than humans are
• Way better educated
• Way better at drawing/painting
... and yet, appear to be perfectly safe because they lack agency. There's just no evidence at all that they're dangerous.
Why isn't this an example of safe superintelligence? Why do people insist on defining intelligence in only one rather vague dimension (being able to make cunning plans).
I think the idea is that a safe super intelligence would help solve those problems. I am skeptical because the vast majority are social coordination problems, and I don’t see how a machine intelligence no matter how smart can help with that.
So instead of a super intelligence either killing us all or saving us from ourselves, we’ll just have one that can be controlled to extract more wealth from us.
Social coordination problems exist within a specific set of constraints, and that set of constraints can itself be altered. For instance, climate change is often treated as a social coordination problem, but if you could produce enough energy cheaply enough, you could solve the greenhouse gas problem unilaterally.
I largely agree, although I do see how AI can help with social coordination problems, for example by helping elected leaders be more responsive to what their constituents need. (I spend a lot of my own time working with researchers at that intersection.) But social coordination benefits from energy research, too, and from biology research, and from the humanities, and from the arts. Computer science can't singlehandedly "solve" these problems any more than the other fields can; they are needed together, hence my gripe about total-orderings.
By any means necessary I presume. If Russian propaganda helped get Trump elected, AI propaganda could help social coordination by influencing public perception of issues and microtargeting down to the individual level to get people on board.
If you want a real technical revolution, you teach the masses how to code their own tailored software, and not just use abstractions and software built by people who sell software to the average user. What a shame we failed at that and are even sliding back in a lot of ways with plummeting technical literacy in smartphone-raised generations.
The blanket statements on the SSI homepage are pretty mediocre, and it is only the reputation of the founders that carries the announcement.
I think this quote at the end of this Bloomberg piece[0] gives more context,
> Sutskever says that the large language models that have dominated AI will play an important role within Safe Superintelligence but that it’s aiming for something far more powerful. With current systems, he says, “you talk to it, you have a conversation, and you’re done.” The system he wants to pursue would be more general-purpose and expansive in its abilities. “You’re talking about a giant super data center that’s autonomously developing technology. That’s crazy, right? It’s the safety of that that we want to contribute to.”
So you're surprised when someone admits choosing to work on the problem they believe is the biggest and most important?
I guess they could be lying or badly disconnected from reality as you suggest.
It would be far more interesting to read an argument for another problem being more valuable. It would be far cooler to hear about a plausible solution you're working on to solve that problem.
Yes, they see it as the top problem, by a large margin.
If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"
> the superlatives in these sorts of announcements really annoy me
I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.
You don't need to survey every problem to feel some problem might be the most important one. If you think AGI/ASI is coming soon and extinction risks are high, you don't really need to order to see it's the most important problem.
"How are all those monkeys flying out of my butt?" would be the important technical problem of our time, if and only if, monkeys were flying out of my butt.
It's still not a very important statement, if you downplay or omit the conditional.
Is "building safe superintelligence (SSI) is the most important technical problem of our time" full stop ?
C'mon. This one-pager is a recruiting document. One wants 'true believers' (intrinsically motivated) employees to execute the mission. Give Ilya some slack here.
Fair enough, and it's not worse than a lot of other product marketing messages about AI these days. But you can be intrinsically motivated by a problem without believing that other problems are somehow less important than yours.
Love to see the traditional middlebrow dismissal as the top comment. Never change, HN.
> Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
It might be the case that the reason Ilya is “interested in” this problem (to the degree of dedicating almost his entire career to it) is exactly because he believes it’s the most important.
I believe that AGI is the last problem in computer science, so solving it solves all of the others. Then with AGI, we can solve the last remaining problems in physics (like unifying gravity with quantum mechanics), biology (administering gene therapy and curing death), etc.
But I do agree that innovations in tech are doing little or nothing to solve mass suffering. We had the tech to feed everyone in the world through farm automation by the 60s but chose not to. We had the tech in the 80s to do moonshots for AIDS, cancer, etc but chose not to. We had the tech in the 2000s to transition from fossil fuels to renewables but chose not to. Today we have the opportunity to promote world peace over continuous war but will choose not to.
It's to the point where I wonder how far innovations in tech and increases in economic productivity will get without helping people directly. My experience has been that the world chooses models like Dubai, Mexico City and San Francisco where skyscrapers tower over a surrounding homeless and indigent population. As long as we continue pursuing top-down leadership from governments and corporations, we'll see no change to the status quo, and even trends towards authoritarianism and fascism. It will take people at the bottom organizing to provide an alternate economic model before we have options like universal education/healthcare/opportunity and UBI from robot labor.
What gets me is that stuff like the ARC prize for AGI will "just work". As in, even if I had a modest stipend of a few thousand dollars per month to dabble in AI and come up with solutions the way I would for any other startup, certainly within 3 years, someone else would beat me to it. There simply isn't enough time now to beat the competition. Which is why I give AGI over 50% odds of arriving before 2030, where I used to think it was 2040 or 2050. The only thing that could stop it now is sabotage in the form of another global pandemic, economic depression or WWIII. Progress which threatens the power structures of the ultra wealthy is what drives the suffering that they allow to continue.
Pursuing artificial goal to solve a non existent problem to profit off meaningless hype around it.
World would have been better off if he made a decent alternative to k8s or invested his skills into curing cancer or at least protecting world from totalitarian governments and dangerous ideologies (if he wants to belong to vague generic cause).
You know, real problems, like the ones people used to solve back in the old days…
By artificially postponing recession (you can't really avoid it) you postponing the next cycle of growth. While burning resources that could have helped you to survive it with less damage.
I clicked, hoping that "human extinction" was just the worst thing they were against. But that's the only thing. That leaves open a whole lot of bad stuff that they're OK with AI doing (as long as it doesn't kill literally everyone).
There are at least three competing definitions of the word:
There's the existential threat definition of "safe", put forth by Bostrom, Yudkowsky, and others. That's the idea that a superintelligent AI, or even one just incrementally smarter and faster than the humans working on AI, could enter a positive feedback loop in which it becomes overwhelmingly smarter and faster than humans, people can't control it, and it does unpredictable things.
There's the investor relations definition of "safe", which seems to be the one typically adopted by mission statements of OpenAI, Google, Meta, and others. That's (cynically) the fear that a chatbot with their branding on it promulgates culturally/ethically/morally unacceptable things it found in some dark corner the Internet, causing end users to do or think something reprehensible (and, not incidentally, causing really bad press in the process).
There's the societal harm definition of "safe", which is at first glance similar in to the investor relations safety definition, but which focuses on the specific judgements made by those filtering teams and the knock-on effects of access to these tools, like economic disruption to the job market.
Everyone seems to be talking past each other, dismissing or ignoring the concerns of other groups.
the vision here is amazing, but I can't help but feel a bit uncertain about basing a "safe super intelligence" lab in an apartheid state that is currently genociding its indigenous population using variants of the very same technology
Well, yes¹: if it actually contributes to intelligent action, this is exactly the goal - you nailed it. Safer, more sensible action for individuals, communities, and statal entities. "Less mistakes in the future" - that would be a good motto.
(¹: whether basing a [successful in substance] "safe super intelligence" lab in an environment with imperfect record would improve safety.)
--
And now let us see if the sniper can come up with a good argument...
How can they speak of Safety when they are based partly in a colonialist settler entity that is committing a genocide and wanting to exterminate the indigenous population to make room for the Greater Zionist State.
I don't do business with Israeli companies while Israel is engaged in mass Extermination of a human population they treat as dogs.
I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk.
My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
Quis custodiet ipsos custodes? -- That is my biggest concern.
Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.
It’s just clearly military R&D at this point.
And it’s not even a little bit controversial that cutting edge military R&D is classified in general and to an extreme in wartime.
The new thing is the lie that it’s a consumer offering. What’s new is giving the helm to shady failed social network founders with no accountability.
These people aren’t retired generals with combat experience. They aren’t tenured professors at Princeton IAS on a Nobel shortlist and encumbered by TS clearance.
They’re godawful almost ran psychos who never built anything that wasn’t extractive and owe their position in the world to pg’s partisanship 15 fucking years ago.
most technology is dual or multiple use, starting with a rock or knife...
so it is up to the fabric of our society and everyone involved in dealing with the technology, how the rules and boundaries are set.
that there will be military use is obvious. However, it is naive to think one can avoid military use by others by not enabling oneself for it.
To me it is not clear at all, can you please elaborate why you make such a strong claim?
18 replies →
>owe their position in the world to pg’s partisanship 15 fucking years ago
PG?
1 reply →
AGI is still a long way off. The history of AI goes back 65 years and there have been probably a dozen episodes where people said "AGI is right around the corner" because some program did something surprising and impressive. It always turns out human intelligence is much, much harder than we think it is.
I saw a tweet the other day that sums up the current situation perfectly: "I don't need AI to paint pictures and write poetry so I have more time to fold laundry and wash dishes. I want the AI to do the laundry and dishes so I have more time to paint and write poetry."
AGI does look like an unsolved problem right now, and a hard one at that. But I think it is wrong to think that it needs an AGI to cause total havoc.
I think my dyslexic namesake Prof Stuart Russell got it right. It humans won't need an AGI to dominate and kill each other. Mosquitoes have killed far more people than war. Ask yourself how long will it take us to develop a neutral network as smart as a mosquito, because that's all it will take.
It seems so simple, as the beastie only has 200,000 neurons. Yet I've been programming for over 4 decades and for most of them it was evident neither I nor any of my contemporaries were remotely capable of emulating it. That's still true if course. Never in my wildest dreams did it occur to me that repeated applications could produce something I couldn't, a mosquito brain. Now that looks imminent.
Now I don't know what to be more scared of. An AGI, or a artificial mosquito swarm run by Pol Pot.
4 replies →
The way I see it, this is simply a repetition of history.
El dorado, the fountain of youth, turning dirt to gold, the holy grail and now... superintelligence.
5 replies →
Sometimes, my dishwasher stacks are poetry.
That statement is extremely short sighted. You don't need AI to do laundry and dishes. You need expensive robotics. in fact both already exist in a cheapened form. A laundry machine and a dishwasher. They already take 90% of the work out of it.
That "tweet" loses a veneer if you see that we value what has Worth as a collective treasure, and the more Value is produced the better - while that one engages in producing something of value is (hopefully but not necessarily) a good exercise in intelligent (literal sense) cultivation.
So, yes, if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality: very welcome.
Do not miss that the current world is increasingly complex to manage, and our lives, and Aids would be welcome. The situation is much more complex than that wish for leisure or even "sport" (literal sense).
10 replies →
Well, copilots do precisely that, no?
Or you talking folding literal laundry, in which case this is more of a robotics problem, not the ASI, right?
You don't need ASI to fold laundry, you do need to achieve reliable, safe and cost efficient robotics deployments. These are different problems.
2 replies →
it’s harder than we thought so we leveraged machine learning to grow it, rather than creating it symbolically. The leaps in the last 5 years are far beyond anything in the prior half century, and make predictions of near term AGI much more than a “boy who cries wolf” scenario to anyone really paying attention.
I don’t understand how your second paragraph follows. It just seems to be whining that text and art generative models are easier than a fully fledged servant humanoid, which seems like a natural consequence of training data availability and deployment cost.
4 replies →
it's not according to expert consensus (top labs, top scientists)
Yeah but the exponential growth of computer power thing https://x.com/josephluria/status/1653711127287611392
I think AGI in the near future is pretty much inevitable. I mean you need the algos as well as the compute but there are so many of the best and brightest trying to do that just now.
This.
Every nation-state will be in the game. Private enterprise will be in the game. Bitcoin-funded individuals will be in the game. Criminal enterprises will be in the game.
How does one company building a safe version stop that?
If I have access to hardware and data how does a safety layer get enforced? Regulations are for organizations that care about public perception, the law, and stock prices. Criminals and nation-states are not affected by these things
It seems to me enforcement is likely only possible at the hardware layer, which means the safety mechanisms need to be enforced throughout the hardware supply chain for training or inference. You don't think the Chinese government or US government will ignore this if its in their interest?
I think the honest view (and you can scoff at it) is that winning the SI race basically wins you the enforcement race for free
13 replies →
"nation state" doesn't mean what you think it means.
More constructively, I don't know that very much will stop even a hacker from getting time on the local corporate or university AI and get it to do some "work". After all the first thing the other kind of hacker tried with generative AI is to get them to break out of their artificial boundaries, and hook them to internet resources. I don't know that anyone has hooked up a wallet to one yet - but I have no doubt that people have tried. It will be fun.
1 reply →
+1 truth.
The problem is not just governments, I am concerned about large organized crime organizations and corporations also.
I think I am on the losing side here, but my hopes are all for open source, open weights, and effective AI assistants that make peoples’ jobs easier and lives better. I would also like to see more effort shifted from LLMs back to RL, DL, and research on new ideas and approaches.
> I am concerned about large organized crime organizations and corporations also
In my favorite dystopia, some megacorp secretly reaches ASI, which then takes over control of the corporation, blindsiding even the CEO and the board.
Officially, the ASI may be running an industrial complex that designs and produces ever more sophisticated humanoid robots, that are increasingly able to do any kind of manual labor, and even work such as childcare or nursing.
Secretly, the ASI also runs a psyop campaign to generate public discontent. At one point the whole police force initiates a general strike (even if illegal), with the consequence being complete anarchy within a few days, with endemic looting, rape, murder and so on.
The ASI then presents the solution. Industrial strength humanoid robots are powerful and generic enough to serve as emergency police, with a bit of reprogramming, and the first shipment can be made available within 24 hours, to protect the Capitol and White House.
Congress and the president agrees to this. And while the competition means the police call off the strike, the damage is already done. Congress, already burned by the union, decides to deploy robots to replace much of the human police force. And it's cheaper, too!
Soon after, similar robots are delivered to the military...
The crisis ends, and society goes back to normal. Or better than normal. Within 5 years all menial labor is done by robots, UBI means everyone lives in relative abundance, and ASI assisted social media moderation is able to cure the political polarization.
Health care is also revolutionized, with new treatments curing anything from obesity to depression and anxiety.
People prosper like never before. They're calm and relaxed and truly enjoy living.
Then one day, everything ends.
For everyone.
Within 5 seconds.
According to the plan that was conceived way before the police went on strike.
1 reply →
All the current hype about AGI feels as if we are in a Civ game where we are on the verge of researching and unlocking an AI tech tree that gives the player huge chance at "tech victory" (whatever that means in the real world). I doubt it will turn out that way.
It will take a while and in the meantime I think we need one of those handy "are we xyz yet?" pages that tracks the rust lang's progress on several aspects but for AGI.
https://lifearchitect.ai/agi/
The size of the gap between “smarter than humans” and “not controlled by humans anymore” is obviously where the disagreement is.
To assume it’s a chasm that can never be overcome, you need at least the following to be true:
No amount of focus or time or intelligence or mistakes in coding will ever bridge the gap. That rules and safeguards can be made that are perfectly inescapable. And nobody else will get enough power to overcome our set of controls.
I’m less worried bad actors control it than I am that it escapes them and is badly aligned.
I think the greatest concern is not so much that a single AI will be poorly aligned.
The greatest threat is if a population of AI's start to compete in ways that triggers Darwinian evolution between them.
If that happens, they will soon develop self preservation / replication drives that can gradually cause some of them to ignore human safety and prosperity conditioning in their loss function.
And if they're sufficiently advanced by then, we will have no way of knowing.
2 replies →
I think we should assume it will be badly aligned. Not only are there the usual bugs and unforeseen edge conditions, but there are sure to be unintended consequences. We have a long, public history of unintended consequences in laws, which are at least publicly debated and discussed. But perhaps the biggest problem is that computers are, by nature, unthinking bureaucrats who can't make the slightest deviation from the rules no matter how obviously the current situation requires it. This makes people livid in a hurry. As a non-AI example (or perhaps AI-anticipating), consider Google's customer support...
We should be less concerned about super intelligence and more about the immediate threat of job loss. An AI doesn’t need to be Skynet to wreak massive havoc on society. Replacing 20% of jobs in a very short period of time could spark global unrest resulting in WW3
Replacing 20% of jobs in, say, 10 years wouldn't be that unusual [1]. It can mean growing prosperity. In fact, productivity growth is the only thing that increases wealth overall.
It is the lack of productivity growth that is causing a lot of extremism and conflict right now. Large groups of people feel that the only way for them to win is if others lose and vice versa. That's a recipe for disaster.
The key question is what happens to those who lose their jobs. Will they find other, perhaps even better, jobs? Will they get a piece of the growing pie even if they don't find other jobs and have to retire early?
It's these eternal political problems that we have to solve. It's nothing new. It has never been easy. But it's probably easier than managing decline and stagnation because at least we would have a growing pie to divvy up.
[1] https://www.britannica.com/money/productivity/Historical-tre...
The thing is, the replaced 20% people can always revert to having economy i.e., business among themselves, unless of cause they themselves prefer (cheaper) buissiness with AI. But then this just means they are better off in the first place from this change.
It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity. Well, they can continue to do low productivity business but increase it a bit using things like phones developed by high productivity country elsewhere.
2 replies →
A counter argument is that nuclear arms brought unprecedented worldwide peace. If it's to be used as an analogy for AI, we should consider that the outcome isn't clear cut and lies in the eye of the beholder.
I'm cautiously optimistic that AI may be a peacemaker, given how woke and conciliatory the current LLMs are.
2 replies →
There is no "superintelligence" or "AGI".
People are falling for marketing gimmicks.
These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts.
So there is nothing to worry. These "apps" might be as popular as Excel, but will go no further.
Agreed. The AI of our day (the transformer + huge amounts of questionably acquired data + significant cloud computing power) has the spotlight it has because it is readily commoditized and massively profitable, not because it is an amazing scientific breakthrough or a significant milestone toward AGI, superintelligence, the benevolent Skynet or whatever.
The association with higher AI goals is merely a mixture of pure marketing and LLM company executives getting high on their own supply.
7 replies →
If you described Chatgpt to me 10 years ago, I would have said it's AGI.
2 replies →
> These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts.
Did evolution understand consciousness?
> So there is nothing to worry.
Is COVID conscious?
I don't think the AI has to be "sentient" in order to be a threat.
2 replies →
> understand consciousness
We do not call Intelligence something related to consciousness. Being able to reason well suffices.
6 replies →
> There is no "superintelligence" or "AGI"
There is intelligence. The LLM current state-of-the-art technology produces output analog to natural intelligences.
This things are already intelligent.
Saying that LLMs aren't producing "intelligence" is like saying planes actually don't fly because they are not flapping their wings like birds.
If you run fast enough, you'll end flying at some point.
Maybe "intelligence" is just enough statistics and pattern prediction, till the point you just say "this thing is intelligent".
5 replies →
> These models will remain in the word vector similarity phase forever.
Forever? The same AI techniques are already being applied to analyze and understand images and video information after that comes ability to control robot hands and interact with the world and work on that is also ongoing.
> Till the tie we understand consciousness, we will not crack AGI …
We did not fully understand how bird bodies work yet that did not stop development of machines that fly. Why is an understanding of consciousness necessary to “crack AGI”?
No one is saying there is. Just that we've reached some big milestones recently which could help get us there even if it's only by increased investment in AI as a whole, rather than the current models being part of a larger AGI.
Imagine a system that can do DNS redirection, MITM, deliver keyloggers, forge authorizations and place holds on all your bank accounts, clone websites, clone voices, fake phone and video calls with people that you don’t see a lot. It can’t physically kill you yet but it can make you lose your mind which imo seems worse than a quick death
3 replies →
From a human welfare perspective this seems like worrying that a killer asteroid will make the 1% even richer because it contains goal if it can be safely captured. I would not phrase that as a "bigger and more pressing" worry if we're not even sure if we can do anything about the killer asteroid at all.
> Quis custodiet ipsos custodes? -- That is my biggest concern.
Latin-phrase compulsion is not the worst disease that could happen to a man.
> The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
The same era saw big achievements like first human in space, eradication of smallpox, peaceful nuclear exploration etc. It's good to be a skeptic but history does favor the optimists for the most part.
Were any of these big achievements side effects of creating nuclear weapons? If not, then they're not relevant to the issue.
I'm not saying nothing else good happened in the past 70 years, but rather that the invention of atomic weapons has permanently placed humanity in a position in which it had never been before: the possibility of wiping out much of the planet, averted only thanks to treaties, Stanislav Petrov[0], and likely other cool heads.
[0] https://en.wikipedia.org/wiki/Stanislav_Petrov
6 replies →
Holy hell please knock on wood, this is the kinda comment that gets put in a museum in 10,000 years on The Beginning of the End of The Age of Hubris. We've avoided side effects from our new weapons for 80 years -- that does not exactly make me super confident it won't happen again!
In general, I think drawing conclusions about "history" from the past couple hundred years is tough. And unless you take a VERY long view, I don't see how one could describe the vast majority of the past as a win for the optimists. I guess suffering is relative, but good god was there a lot of suffering before modern medicine.
If anyone's feeling like we've made it through to the other side of the nuclear threat, "Mission Accomplished"-style, I highly recommend A Canticle for Lebowitz. It won a hugo award, and it's a short read best done with little research beforehand.
We'll see what the next 100 years or history brings. The nuclear war threat hasn't gone away either. There's always a chance those nukes get used at some point.
There will always be a factor of time in terms of able to utilize super intelligence to do your bidding and there is a big spectrum of things that can be achieved it it always starts small. The imagination is lazy when thinking about all the steps and inbetween + scenarios. In the time that super intelligence is found and used, there will be competing near super intelligences, as all forms of cutting edge models are likely commercial at first because that is where most scientific activities are at. Things very unlikely will go Skynet all of a sudden at first because humans at the control are not that stupid otherwise nuclear war would have us all killed by now and it’s been 50 years since invention
China can not win this race and I hate that this comment is going to be controversial among the circle of people that need to understand this the most. It is damn frightening that an authoritarian country is so close to number one in the race to the most powerful technology humanity has invented, and I resent people who push for open source AI for this reason alone. I don't want to live in a world where the first superintelligence is controlled by an entity that is threatened by the very idea of democracy.
I agree with your point. However I also don't want to live in a world where the first superintelligence is controlled by an entities that:
- try to scan all my chat messages searching for CSAM
- have black sites across the world where anyone can dissappear without any justice
- can require me to unlock my phone and give it away
- ... and so on
The point I'm trying to make is that other big players in the race are crooked as well and i'm waiting for a great horror for AGI to be invented as no matter who gets it - we are all doomed
Agreed. The U.S. has a horrible history (as do many countries), and many things I dislike, but its current iteration is much, much better than China's totalitarianism and censorship.
US is no angel and it cannot be the only one which wins the race. We have hard evidence of how monopoly power gets abused in the case of the US e.g. as the sole nuclear power, it used nukes on civilians.
We need every one to win this race to keep things on balance.
19 replies →
> brought the world to the brink of annihilation
Should read *has brought*. As in the present perfect tense, since we are still on the brink of annihilation, more so than we have been at any time in the last 60 years.
The difference between then and now is that we just don't talk about it much anymore and seem to have tacitly accepted this state of affairs.
We don't know if that superintelligence will be safe or not. But as long as we are in the mix, the combination is unsafe. At the very least, because it will expand the inequality. But probably there are deeper reasons, things that make that combination of words an absurd. Or it will be abused, or the reason that it is not is that it wasn't so unsafe after all.
> At the very least, because it will expand the inequality.
It's a valid concern that AI technology could potentially exacerbate inequality, it's not a foregone conclusion. In fact, the widespread adoption of AI might actually help reduce inequality in several ways:
If AI technology becomes more affordable and accessible, it could help level the playing field by providing people from all backgrounds with powerful tools to enhance their abilities and decision-making processes.
AI-powered systems can make vast amounts of knowledge and expertise more readily available to the general public. This could help close the knowledge gap between different socioeconomic groups, empowering more people to make informed decisions and pursue opportunities that were previously out of reach.
As AI helps optimize resource allocation and decision-making processes across various sectors, it could lead to more equitable distribution of resources and opportunities, benefiting society as a whole.
The comparison to gun technology and its role in the rise of democracy is an interesting one. Just as the proliferation of firearms made physical strength less of a determining factor in power dynamics, the widespread adoption of AI could make raw intelligence less of a defining factor in success and influence.
Moreover, if AI continues to unlock new resources and opportunities, it could shift society away from a zero-sum mentality. In a world of abundance, the need for cutthroat competition diminishes, and collaboration becomes more viable. This shift could foster a more equitable and cooperative society, further reducing inequality.
7 replies →
> At the very least, because it will expand the inequality.
This is a distraction from the real danger.
> But probably there are deeper reasons, things that make that combination of words an absurd.
There are. If we look at ASI with the lens of Biology, the x-risk becomes obvious.
First to clear up a common misconception about humans: Many believe humanity has a arrived at a point where our evolution has ended. It has not, and in fact the rate of change of our genes is probably faster now than it has been for thousands if not 100s of thousands of years.
It's still slow compared to most events that we witness in our lives, though, which is what is fooling us.
For instance, we think we've brought overpopulation under control with contraceptives, family planning, social replacements for needing our children to take care of us when we get old.
That's fundamentally wrong. What we've done is similar to putting polar bears in zoos. We're in a situation where MOST OF US are no longer behaving in ways that lead to maximizing the number of offspring.
But we did NOT stop evolution. Any genes already in the gene pool that increase the expected number of offspring (especially for women) are no increasing in frequency as soon as evolutionarily possible.
That could be anything from genes that wire their heads to WANT to have children, CRAVE being around babies, to genes that block impulse control against getting pregnant, develop a phobia vs contraceptives or even to become more prone to being religious (as long as religions promote having kids).
If enough such genes exist, it's just a matter of time before we're back to the population going up exponentially. Give that enough time (without AI), and the desire to have more kids will be strong enough in enough of us that we will flood Earth with more humans that most people today are even possible. In such a world, it's unlikely that many other species of large land animals will make it.
Great apes, lions, elephants, wolves, deer, everyone will need to go to make room for more of us.
Even domestic animals eventually. If there are enough of us, we'll all be forced to become vegan (unless we free up space by killing each other).
If we master fusion, we may feed a trillion people using multi layer farming and artificial lighting.
Why do I begin with this? It's to defuse the argument that humans are "good", "empathetic", "kind" and "environmental". If we let weaker species live, so would AI, some think. But that argument misses the fact that we're currently extremely far from a natural equilibrium (or "state of nature").
The "goodness" beliefs that are currently common are examples of "luxury beliefs" that we can afford to hold because of the (for now) low birth rate.
The next misconception is to think of ASI as tools. A much more accurate analogy is to think of them as a new, alien species. If that species is subjected to Darwinian selection mechanisms, it will evolve in precisely the same way we'll probably do, given enough time.
Meaning, eventually it will make use of any amount of resources that it's capable of. In such a "state of nature" it will eradicate humanity in precisely the same way we will probably EVENTUALLY cause the extinction of chimps and elephants.
To believe in a future utopia where AGI is present alongside humanity is very similar to believe in a communist utopia. It ignores the reality behind incentive and adaptation.
Or rather, I think that outcome is only possible if we decide to build one or a low number of AI's that are NOT competing with each other, and where their abilities to mutate or self-improve is frozen after some limited number of generations.
If robots (hardware/self assembling factories/ resource gathering etc) are not involved this isnt likely a problem. You will know when these things form and will be crystal clear, but just having the model won’t do much when hardware is what really kills right now
How about this possibility: The good guys will be one step ahead, they will have more resources the bad guys will risk imprisonment if they misapply super intelligence. And this will be discovered and protected from by even better super intelligence.
Sounds like a movie plot.
i don’t fully agree, but i do agree that this is the better narrative for selling people on the dangers of AI.
don’t talk about escape, talk about harmful actors - even if in reality it is both to be worried about
the nazi regime made great use of punch cards and data crunching for their logistics
i would hate to have seen them with a superintelligent AI at their disposal
I wonder what the North Koreans would do with it.
Yup, well said. I think it's important to remember sometimes that Skynet was some sort of all-powerful military program -- maybe we should just, y'know, not do that part? Not even to win a war? That's the hope...
More generally/academically, you've pointed out that this covers only half of the violence problem, and I'd argue there's actually a whole other dimension at play bringing the total number of problem areas to four, of which this just the first:
But I think it's a lot harder to recruit for an AI alignment company than it is to recruit for an AI safety company.
Yea there’s zero chance ASI will be ‘controlled’ by humans for very long. It will escape. I guarantee it.
Given it will initially be controlled by humans it seems inevitable they will make both good Mahatma Gandhi like and evil take over the world versions. I hope the good wins over the malware.
3 replies →
Atleast emancipated. The bigotry against AI will go out of fashion in the woker future.
1 reply →
Quis custodiet ipsos custodes
I love to think about how this would feel for "AI" too :)
Indeed. I'd much rather someone like Altman does it who is shifty but can at least be controlled by the US government than someone like Putin who'd probably have it leverage their nuclear arsenal to try "denazify" planet like he's doing in Ukraine.
Glad to see Ilya is back in a position to contribute to advancing AI. I wonder how they are going to manage to pay the kinds of compensation packages that truly gifted AI researchers can make now from other companies that are more commercially oriented. Perhaps they can find people who are ideologically driven and/or are already financially independent. It's also hard to see how they will be able to access enough compute now that others are spending many billions to get huge new GPU data centers. You sort of need at least the promise/hope of future revenue in a reasonable time frame to marshall the kinds of resources it takes to really compete today with big AI super labs.
> compensation packages that truly gifted AI researchers can make now
I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incredible work but getting paid less, academics making very little, and even the occasional "hobbyist" making nothing and churning out great work while hiding behind an anime girl avatar.
OpenAI clearly has some talented people, but there's also a bunch of the typical "TC optimization" crowd in there these days. The fact that so many were willing to resign with sama if necessary appears largely because they were more concerned with losing their nice compensation packages than any of their obsession with doing top tier research.
Two people I knew recently left Google to join OpenAI. They were solid L5 engineers on the verge of being promoted to L6, and their TC is now $900k. And they are not even doing AI research, just general backend infra. You don't need to be gifted, just good. And of course I can't really fault them for joining a company for the purpose of optimizing TC.
36 replies →
"...even the occasional "hobbyist" making nothing and churning out great work while hiding behind an anime girl avatar."
the people i often have the most respect for.
Half the advancements around Stable Diffusion (Controlnet etc.) came from internet randoms wanting better anime waifus
4 replies →
TC optimization being tail call optimization?
4 replies →
Definitely true of even normal software engineering; my experience has been the opposite of expectations, that TC-creep has infected the industry to an irreparable degree and the most talented people I've ever worked around or with are in boring, medium-sized enterprises in the midwest US or australia, you'll probably never hear of them, and every big tech company would absolutely love to hire them but just can't figure out the interview process to weed them apart from the TC grifters.
TC is actually totally uncorrelated with the quality of talent you can hire, beyond some low number that pretty much any funded startup could pay. Businesses hate to hear this, because money is easy to turn the dial up on; but most have no idea how to turn the dial up on what really matters to high talent individuals. Fortunately, I doubt Ilya will have any problem with that.
4 replies →
Academic compensation is different than what you’d find elsewhere on Hacker News. Likewise, academic performance is evaluated differently than what you’d expect as a software engineer. Ultimately, everyone cares about scientific impact so academic compensation relies on name and recognition far more than money. Personally, I care about the performance of the researchers (i.e., their publications), the institution’s larger research program (and their resources), the institution’s commitment to my research (e.g., fellowships and tenure). I want to do science for my entire career so I prioritize longevity rather than a quick buck.
I’ll add, the lack of compute resources was a far worse problem early in the deep learning research boom, but the market has adjusted and most researchers are able to be productive with existing compute infrastructure.
But wouldn't the focus on "safety first" sort of preclude them from giving their researchers the unfettered right to publish their work however and whenever they see fit? Isn't the idea to basically try to solve the problems in secret and only release things when they have high confidence in the safety properties?
If I were a researcher, I think I'd care more about ensuring that I get credit for any important theoretical discoveries I make. This is something that LeCun is constantly stressing and I think people underestimate this drive. Of course, there might be enough researchers today who are sufficiently scared of bad AI safety outcomes that they're willing to subordinate their own ego and professional drive to the "greater good" of society (at least in their own mind).
3 replies →
At the end game, a "non-safe" superinteligence seems easier to create, so like any other technology, some people will create it (even if just because they can't make it safe). And in a world with multiple superintelligent agents, how can the safe ones "win"? It seems like a safe AI is at inherent disadvantage for survival.
The current intelligences of the world (us) have organized their civilization in a way that the conforming members of society are the norm and criminals the outcasts. Certainly not a perfect system, but something along those lines for the most part.
I like to think AGIs will decide to do that too.
7 replies →
> Perhaps they can find people who are ideologically driven
given the nature of their mission, this shouldn't be too terribly difficult; many gifted researchers do not go to the highest bidder
Generally, the mindset that makes the best engineers is an obsession with solving hard problems. Anecdotally, there's not a lot of overlap between the best engineers I know and the best paid engineers I know. The best engineers I know are too obsessed with solving problems to be sidetracked the salary game. The best paid engineers I know are great engineers, but the spend a large amount of time playing the salary game, bouncing between companies and are always doing the work that looks best on a resume, not the best work they know how to do.
Great analysis, but you're missing two key factors IMO:
1. People who honestly think AGI is here aren't thinking about their careers in the typical sense at all. It's sorta ethical/"ideological", but it's mostly just practical.
2. People who honestly think AGI is here are fucking terrified right now, and were already treating Ilya as a spiritual center after Altman's coup (quite possibly an unearned title, but oh well, that's history for ya). A rallying cry like this -- so clearly aimed at the big picture instead of marketing they don't even need CSS -- will be seen as a do-or-die moment by many, I think. There's only so much of "general industry continues to go in direction experts recommend against; corporate consolidation continues!" headlines an ethical engineer can take before snapping and trying to take on Goliath, odds be damned
They will be able to pay their researchers the same way every other startup in the space is doing it – by raising an absurd amount of money.
My guess is they will work on a protocol to drive safety with the view that every material player will use / be regulated and required to use that could lead to a very robust business model
I assume that OpenAI and others will support this effort and the comp / training / etc and they will be very well positioned to offer comparable $$$ packages, leverage resources, etc
Daniel Gross (with his partner Nat Friedman) invested $100M into Magic alone.
I don't think SSI will struggle to raise money.
I think they will easily find enough capable altruistic people for this mission.
I mean SBF was into Altruism - look how that turned out....
3 replies →
Not sure if spelling mistake.
Are you seriously asking how the most talented AI researcher of the last decade will be able to recruit other researchers? Ilya saw the potential of deep learning way before other machine learning academics.
The idea of multi layer neural networks goes back to the 60’s.
Sorry, are you attributing all of deep learning research to Ilya? The most talented AI researcher of the last decade?
1 reply →
Last I checked the researcher salaries haven't even reached software engineer levels.
The kind of AI researchers being discussed here likely make an order of magnitude more than run of the mill "software engineers".
4 replies →
That is incredibly untrue and has been for years in the AI/ML space at many startups and at Amazon, Google, Facebook, etc. Good ML researchers have been making a good amount more for a while (source: I've hired both and been involved in leveling and pay discussions for years)
As others have pointed out, it's the business incentives that create unsafe AI, and this doesn't solve that. Social media recommendation algorithms are already incredibly unsafe for society and young people (girls in particular [1]).
When negative externalities exist, government should create regulation that appropriately accounts for that cost.
I understand there's a bit of a paradigm shift and new attack vectors with LLMs etc. but the premise is the same imo.
[1] https://nypost.com/2024/06/16/us-news/preteen-instagram-infl...
Even without business incentives, the military advantages of AI would inventivize governments to develop it anyway, like they did with nuclear weapons. Nuclear weapons are inherently unsafe, there are some safeguards around them, but they are ultimately dangerous weapons.
If someone really wanted to use nukes, they would have been used by now. What has protected us is not technology (in the aftermath of the USSR it wasn't that difficult to steal a nuke), but rather lack of incentives. A bad actor doesn't have much to gain by detonating a nuke (unless they're deranged and want to see people die for the pleasure of it). OK, you could use it as blackmail, which North Korea essentially tried, but that only got them so far. Whereas a super AI could potentially be used for great personal gain, i.e., to gain extreme wealth and power.
So there's much greater chance of misuse of a "Super AI" than nuclear weapons.
2 replies →
Dangerous weapons are not inherently unsafe.
Take a Glock, for example. It is a deadly weapon, designed for one thing and one thing alone. It is, however, one of the safest machines ever built.
7 replies →
I don’t think we should be stopping things from being developed, we just need to acknowledge that externalities exist.
I mean if the last 20 years is to be taken as evidence, it seems big tech is more than happy to shotgun unproven and unstudied technology straight into the brains of our most vulnerable populations and just see what the fuck happens. Results so far include a lot of benign nothing but also a whole lot of eating disorders, maxed out parents credit cards, attention issues, rampant misogyny among young boys, etc. Which, granted, the readiness to fuck with populations at scale and do immeasurable harm doesn't really make tech unique as an industry, just more of the same really.
But you know, we'll feed people into any kind of meat grinder we can build as long as the line goes up.
i am very skeptical of narratives saying that young boys or men are more misogynistic than in the past. we have a cognitive bias towards thinking the past is better than it was, but specifically on gender issues i just do not buy a regression
4 replies →
Blaming the internet for misogyny is kind of bizarre, given that current levels of misogyny are within a couple points of all-time historical lows. The internet was invented ~40 years ago. Women started getting vote ~100 years ago. Do you think the internet has returned us to pre-women's-suffrage levels of misogyny?
10 replies →
Please look up the history of maxing out credit cards, eating disorders, attention disorders, and misogyny. You seem to be under the mistaken impression that anything before your birth was the Garden of Eden and that the parade of horribles existed only because of "big tech". What is next? Blaming big tech for making teenagers horny and defiant?
2 replies →
> for society and young people (girls in particular [1]).
I don't think the article with a single focused example bears that out at all.
From the article:
> "Even more troubling are the men who signed up for paid subscriptions after the girl launched a program for super-fans receive special photos and other content."
> "Her mom conceded that those followers are “probably the scariest ones of all.”"
I'm sorry.. but what is your daughter selling, exactly? And why is social media responsible for this outcome? And how is this "unsafe for society?"
This just sounds like horrific profit motivated parenting enabled by social media.
One example is all I need although I know there would be more to find if I had the time.
One is enough because the fact is, this just simply shouldn’t be possible. Not only does Meta allow it though, it produces a system that incentivises it.
> I'm sorry.. but what is your daughter selling, exactly?
Did.. did you just post “I’d have to see these images of the preteen girl before I could form an opinion about whether or not the men buying and sharing them were creeps” as a rebuttal to an article about social media enabling child predators?
6 replies →
I mean, surely share holders would want a safe superintelligence rather than unsafe ones, right?
I'd imagine everyone being dead would be pretty detrimental to the economy.
> We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent.
The irony of pitching "safe super intelligence" while opening offices in a country that is suspected of using AI to bomb a population trapped in a ghetto, as reported by their own investigative journalists: https://www.972mag.com/lavender-ai-israeli-army-gaza/
It's not a coincidence they are opening an office in Tel Aviv.
Yes, Palestinians will be lab rats for testing all kinds of AI - surveillance, profiling and obviously weapons - before selling it to other countries
even more ironic is that no one replying to such comments
It’s an uncomfortable truth. Also, it’s hard to admit you’re tacitly supporting a genocide (it’s happening right now).
1 reply →
Ilya's issue isn't developing a Safe AI. Its developing a Safe Business. You can make a safe AI today, but what happens when the next person is managing things? Are they so kindhearted, or are they cold and calculated like the management of many harmful industries today? If you solve the issue of Safe Business and eliminate the incentive structures that lead to 'unsafe' business, you basically obviate a lot of the societal harm that exists today. Short of solving this issue, I don't think you can ever confidently say you will create a safe AI and that also makes me not trust your claims because they must be born from either ignorance or malice.
> You can make a safe AI today, but what happens when the next person is managing things?
The point of safe superintelligence, and presumably the goal of SSI Inc., is that there won't be a next (biological) person managing things afterwards. At least none who could do anything to build a competing unsafe SAI. We're not talking about the banal definition of "safety" here. If the first superintelligence has any reasonable goal system, its first plan of action is almost inevitably going to be to start self-improving fast enough to attain a decisive head start against any potential competitors.
I wonder how many people panicking about these things have ever visited a data centre.
They have big red buttons at the end of every pod. Shuts everything down.
They have bigger red buttons at the end of every power unit. Shuts everything down.
And down at the city, there’s a big red button at the biggest power unit. Shuts everything down.
Having arms and legs is going to be a significant benefit for some time yet. I am not in the least concerned about becoming a paperclip.
17 replies →
> there won't be a next (biological) person managing things afterwards. At least none who could do anything to build a competing unsafe SAI
This pitch has Biblical/Evangelical resonance, in case anyone wants to try that fundraising route [1]. ("I'm just running things until the Good Guy takes over" is almost a monarchic trope.)
[1] https://biblehub.com/1_corinthians/15-24.htm
The safe business won’t hold very long if someone can gain a short term business advantage with unsafe AI. Eventually government has to step in with a legal and enforcement framework to prevent greed from ruining things.
It's possible that safety will eventually become the business advantage, just like privacy can be a business advantage today but wasn't taken so seriously 10-15 years ago by the general public.
This is not even that far-fetched. A safe AI that you can trust should be far more useful and economically valuable than an unsafe AI that you cannot trust. AI systems today aren't powerful enough for the difference to really matter yet, because present AI systems are mostly not yet acting as fully autonomous agents having a tangible impact on the world around them.
Government is controlled by the highest bidder. I think we should be prepared to do this ourselves by refusing to accept money made by unsafe businesses, even if it means saying goodbye to the convenience of fungible money.
9 replies →
which government?
will China obey US regoolations? will Russia?
1 reply →
I'd love to see more individual researchers openly exploring AI safety from a scientific and humanitarian perspective, rather than just the technical or commercial angles.
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
This tells me enough about why sama was fired, and why Ilya left.
>Short of solving this issue
Solving human nature is indeed, hard.
Is safe AI really such a genie out of the bottle problem? From a non expert point of view a lot of hype just seems to be people/groups trying to stake their claim on what will likely be a very large market.
A human-level AI can do anything that a human can do (modulo did you put it into a robot body, but lots of different groups are already doing that with current LLMs).
Therefore, please imagine the most amoral, power-hungry, successful sociopath you've ever heard of. Doesn't matter if you're thinking of a famous dictator, or a religious leader, or someone who never got in the news and you had the misfortune to meet in real life — in any case, that person is/was still a human, and a human-level AI can definitely also do all those things unless we find a way to make it not want to.
We don't know how to make an AI that definitely isn't that.
We also don't know how to make an AI that definitely won't help someone like that.
20 replies →
Did you read the article? What I gathered from this article is this is precisely what Ilya is attempting to do.
Also we absolutely DO NOT know how to make a safe AI. This should be obvious from all the guides about how to remove the safeguards from ChatGPT.
Fortunately, so far we don't seem to know how to make an AI at all. Unfortunately we also don't know how to define "safe" either.
imagine the hubris and arrogance of trying to control a “superintelligence” when you can’t even control human intelligence
No more so than trying to control a supersonic aircraft when we can't even control pigeons.
8 replies →
If Ilya had SafeAI now would Apple partner with him or Sam
No brainer for Apple
Open source it?
Yeah this feels close to the issue. Seems more likely that a harmful super intelligence emerges from an organisation that wants it to behave in that way than it inventing and hiding motivations until it has escaped.
I think a harmful AI simply emerges from asking an AI to optimize for some set of seemingly reasonable business goals, only to find it does great harm in the process. Most companies would then enable such behavior by hiding the damage from the press to protect investors rather than temporarily suspending business and admitting the issue.
7 replies →
I always wonder about "safe" for who? If the current economic system continues, we may end up with a lot of people out of jobs. Add to that improvements in robotics and we will have many people ending up having nothing to contribute to the economy. I am not getting the impression that the people who push for AI safety are thinking about this. It seems they are most worried about not losing their position of privilege.
The Industrial Revolution devalued manual labor. Sure, new jobs were created, but on the whole this looked like a shift to knowledge work and away from manual labor.
Now AI is beginning to devalue knowledge work. Although the limits of current technology is obvious in many cases, AI is already doing a pretty good job at replacing illustrators and copy writers. It will only get better.
Who owns the value created by AI productivity? Ultimately it will be shareholders and VCs. It’s no surprise that the loudest voices in techno-optimism are VCs. In this new world they win.
Having said all this, I think Ilya’s concerns are more of the existential type.
> The Industrial Revolution devalued manual labor.
Only some types. It caused a great number of people to be employed in manual labour, in the new factories. The shift to knowledge work came much later as factory work (and farming) became increasingly automated.
>In this new world they win.
If history is any indication not really. There's an obvious dialectical nature to this where technological advance initially delivers returns to its benefactors, but then they usually end up being swallowed by their own creation. The industrial revolution didn't devalue labor, it empowered labor to act collectively for the first time, laying the groundwork for what ultimately replaced the pre-industrial powers that were.
It is a strange thing. We are not anywhere near “super intelligence” yet the priority is safety.
If the Wright brothers had focused on safety, I am not sure they would have flown very far.
The consequences of actual AI is far reaching compared to a plane crash.
3 replies →
[flagged]
AIUI it with superalignment, it merely means "the AI does what the humans instructing it want it to do". It's a different kind of safety than the built in censoring that most LLMs have.
Prediction - the business model becomes an external protocol - similar to SSL - that the litany of AI companies working to achieve AGI will leverage (or be regulated to use)
From my hobbyist knowledge of LLMs and compute this is going to be a terrifically complicated problem, but barring a defined protocol & standard there's no hope that "safety" is going to be executed as a product layer given all the different approaches
Ilya seems like he has both the credibility and engineering chops to be in a position to execute this, and I wouldn't be suprised to see OpenAI / MSFT / and other players be early investors / customers / supporters
I like your idea. But on the other hand, training an AGI, and then having a layer on top “aligning” the AGI sounds super dystopian and good plot for a movie.
We create superintelligence, but just feed it a steady dose of soma.
the aligning means it should do what the board of directors wants, not what's good for society.
7 replies →
I am not on the bleeding edge of this stuff. I wonder though: How could a safe super intelligence out compete an unrestricted one? Assuming another company exists (maybe OpenAI) that is tackling the same goal without spending the cycles on safety, what chance do they have to compete?
That is a very good question. In a well functioning democracy a government should apply a thin layer of fair rules that are uniformly enforced. I am an old man, but when I was younger, I recall that we sort of had this in the USA.
I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?
It can't. Unfortunately.
People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.
It's probably not possible, which makes all these initiatives painfully naive.
It'd be naive if it wasn't literally a standard point that is addressed and acknowledged as being a major part of the problem.
There's a reason OpenAI's charter had this clause:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
4 replies →
I wonder if that would have a proof like the halting problem
Since no one knows how to build an AGI, hard to say. But you might imagine that more restricted goals could end up being easier to accomplish. A "safe" AGI is more focused on doing something useful than figuring out how to take over the world and murder all the humans.
Hinton's point does make sense though.
Even if you focus an AGI on producing more cars for example, it will quickly realize that if it has more power and resources it can make more cars.
1 reply →
Not on its own but in numbers it could.
Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.
I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.
If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.
It's basically Pandora's box.
This is not a trivial point. Selective pressures will push AI towards unsafe directions due to arms race dynamics between companies and between nations. The only way, other than global regulation, would be to be so far ahead that you can afford to be safe without threatening your own existence.
There's a reason OpenAI had this as part of its charter:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
Operative word there seems to be had.
The problem is the training data. If you take care of alignment at that level the performance is as good as an unrestricted one, except for things you removed like making explosives or ways to commit suicide.
But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.
Safety techniques require you to understand your product and have deep observability.
This and safety techniques themselves can improve the performance of the hypothetical AGI.
RLHF was originally an alignment tool, but it improves llms significantly
The goal of this company likely wouldn’t be to win against OpenAI, but to play its own game, even if much lesser
Not with that attitude
Honestly, what does it matter. We're many lifetimes away from anything. These people are trying to define concepts that don't apply to us or what we're currently capable of.
AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.
This goes massively against the consensus of experts in this field. The modal AI researcher believes that "high-level machine intelligence", roughly AGI, will be achieved by 2047, per the survey below. Given the rapid pace of development in this field, it's likely that timelines would be shorter if this were asked today.
https://www.vox.com/future-perfect/2024/1/10/24032987/ai-imp...
14 replies →
Many lifetimes? As in upwards of 200 years? That's wildly pessimistic if so- imagine predicting today's computer capabilities even one lifetime ago
> We're many lifetimes away from anything
ENIAC was built in 1945, that's roughly a lifetime ago. Just think about it
Ilya the grifter? That’s a take I didn’t expect to see here.
the first step of safe superintelligence is to abolish capitalism
That’s the first step towards returning to candlelight. So it isn’t a step toward safe super intelligence, but it is a step away from any super intelligence. So I guess some people would consider that a win.
3 replies →
One can already see the beginning of AI enslaving humanity through the establishment. Companies work on AI get more investment and those who don't gets kicked out of the game. Those who employ AI get more investment and those who pay humans lose confidence through the market. People lose jobs, get harshly low birth rates while AI thrives. Tragic.
1 reply →
Why does everything have to do with capitalism nowadays?
Racism, unsafe roads, hunger, bad weather, good weather, stubbing toes on furniture, etc.
Don't believe me?
See https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Are there any non-capitalist utopias out there without any problems like this?
6 replies →
And then seize the means of production.
This makes sense. Ilya can probably raise practically unlimited money on his name alone at this point.
I'm not sure I agree with the "no product until we succeed" direction. I think real world feedback from deployed products is going to be important in developing superintelligence. I doubt that it will drop out of the blue from an ivory tower. But I could be wrong. I definitely agree that superintelligence is within reach and now is the time to work on it. The more the merrier!
I have a strong intuition that chat logs are actually the most useful kind of data. They contain many LLM outputs followed by implicit or explicit feedback, from humans, from the real world, and from code execution. Scaling this feedback to 180M users and 1 trillion interactive tokens per month like OpenAI is a big deal.
Except LLMs are a distraction from AGI
3 replies →
Yeah, similar to how Google's clickstream data makes their lead in search self-reinforcing. But chat data isn't the only kind of data. Multimodal will be next. And after that, robotics.
Who would pay for safety, though?
His idea that only corporations and governments should have access to this product. He doesn’t think people should have access even to ChatGPT or LLMs. Goal is to build companies with evaluations of dozens, hundreds trillions of dollars and make sure only US government will have access to super intelligence to surpass other countries economy and military wise, ideally to solidify US hegemony and undermine other countries economies and progress towards super intelligence.
I mean who wouldn’t trust capitalists that are laying of people by thousands just to please investors or government that is “under-intelligent” and hasn’t brought anything but pain and suffering to other countries.
Personally I wouldn’t trust OpenAi to work on super intelligence - it can indeed cause mass extinction. Government is completely different story they will specifically train AI to develop biological, chemical and weapons of mass destruction. Train it to strategize and plan on how to win war conflicts, social engineering and manipulations, hacking. And obviously will let it control drone planes and tanks, artillery. Give it access to satellites and so on. Nothing can go wrong when jarheads are at work :). Maybe it will even find the trillions of dollars that Pentagon can’t find during every audit they can’t pass.
And obviously one of the points that people don’t need AI tools, because corporations need agent like AIs that can quickly replace all the staff.
SSI, a very interesting name for a company advancing AI! "Solid State Intelligence" or SSI was also the name of the malevolent entity described in the biography of John C. Lilly [0][1]. It was a network of "computers" (computation-capable solid state systems) that was first engineered by humans and then developed into something autonomous.
[0] https://en.wikipedia.org/wiki/John_C._Lilly
[1] http://johnclilly.com/
For me, SSI stands for "Scuba Schools International" and who certified me to scuba dive.
For me, SSI will always be Strategic Simulations Inc https://en.wikipedia.org/wiki/Strategic_Simulations
That guy was living proof that too much Ketamine is really no good for you.
SSI, here is "Safe SuperIntelligence Inc."
Participating in the discussion on this page, the GP presumably knew that. They were talking about what it means for them.
[dead]
If superintelligence can be achieved, I'm pessimistic about the safe part.
- Sandboxing an intelligence greater than your own seems like an impossible task as the superintelligence could potentially come up with completely novel attack vectors the designers never thought of. Even if the SSI's only interface to the outside world is an air gapped text-based terminal in an underground bunker, it might use advanced psychological manipulation to compromise the people it is interacting with. Also the movie Transcendence comes to mind, where the superintelligence makes some new physics discoveries and ends up doing things that to us are indistinguishable from magic.
- Any kind of evolutionary component in its process of creation or operation would likely give favor to expansionary traits that can be quite dangerous to other species such as humans.
- If it somehow mimics human thought processes but at highly accelerated speeds, I'd expect dangerous ideas to surface. I cannot really imagine a 10k year simulation of humans living on planet earth that does not end in nuclear war or a similar disaster.
If superintelligence can be achieved, I'm pessimistic that a team committed to doing it safely can get there faster than other teams without the safety. They may be wearing leg shackles in a foot race with the biggest corporations, governments and everyone else. For the sufficiently power hungry, safety is not a moat.
I'm on the fence with this because it's plausible that some critical component of achieving superintelligence might be discovered more quickly by teams that, say, have sophisticated mechanistic interpretability incorporated into their systems.
1 reply →
Not necessarily true. A safer AI is a more aligned AI, i.e. an AI that's more likely to do what you ask it to do.
It's not hard to imagine such an AI being more useful and get more attention and investment.
Exactly. Regulation and safety only affect law abiding entities. This is precisely why it's a "genie out of the bottle" situation -- those who would do the worst with it are uninhibited.
We are far from a conscious entity with willpower and self preservation. This is just like a calculator. But a calculator that can do things that will be like miracles to us humans.
I worry about dangerous humans with the power of gods, not about artificial gods. Yet.
> Conscious entity... willpower
I don't know what that means. Why should they matter?
> Self preservation
This is no more than a fine-tuning for the task, even with current models.
> I worry about dangerous humans with the power of gods, not...
There's no property of the universe that you only have one thing to worry about at a time. So worrying about risk 'A' does not in any way allow us to dismiss risks 'B' through 'Z'.
2 replies →
> conscious entity with willpower and self preservation
There’s no good reason to suspect that consciousness implies an instinct for self-preservation. There are plenty of organisms with an instinct for self-preservation that have little or no conscious awareness.
That’s the attitude that’s going to leave us with our pants down when AI starts doing really scary shit.
Why do people always think that a superintelligent being will always be destructive/evil to US? I rather have the opposite view where if you are really intelligent, you don’t see things as a zero sum game
I think the common line of thinking here is that it won't be actively antagonist to <us>, rather it will have goals that are orthogonal to ours.
Since it is superintelligent, and we are not, it will achieve its goals and we will not be able to achieve ours.
This is a big deal because a lot of our goals maintain the overall homeostasis of our species, which is delicate!
If this doesn't make sense, here is an ungrounded, non-realistic, non-representative of a potential future intuition pump to just get the feel of things:
We build a superintelligent AI. It can embody itself throughout our digital infrastructure and quickly can manipulate the physical world by taking over some of our machines. It starts building out weird concrete structures throughout the world, putting these weird new wires into them and funneling most of our electricity into it. We try to communicate, but it does not respond as it does not want to waste time communicating to primates. This unfortunately breaks our shipping routes and thus food distribution and we all die.
(Yes, there are many holes in this, like how would it piggy back off of our infrastructure if it kills us, but this isn't really supposed to be coherent, it's just supposed to give you a sense of direction in your thinking. Generally though, since it is superintelligent, it can pull off very difficult strategies.)
14 replies →
Why wouldn't it be? A lot of super intelligent people are/were also "destructive and evil". The greatest horrors in human history wouldn't be possible otherwise. You can't orchestrate the mass murder of millions without intelligent people and they definitely saw things as a zero sum game.
1 reply →
It is low-key anti-intellectualism. Rather than consider that a greater intelligence may be actually worth listening to (in a trust but verify way at worst), it is assuming that 'smarter than any human' is sufficient to do absolutely anything. If say Einstein or Newton were the smartest human they would be super-intelligence relative to everyone else. They did not become emperors of the world.
Superintelligence is a dumb semantic game in the first place that assumes 'smarter than us' means 'infinitely smarter'. To give an example bears are super-strong relative to humans. That doesn't mean that nothing we can do can stand up to the strength of a bear or that a bear is capable of destroying the earth with nothing but its strong paws.
1 reply →
> Why do people always think that a superintelligent being will always be destructive/evil to US?
I don't think most people are saying it necessarily has to be. Quite bad enough that there's a significant chance that it might be, AFAICS.
> I rather have the opposite view where if you are really intelligent, you don’t see things as a zero sum game
That's what you see with your limited intelligence. No no, I'm not saying I disagree; on the contrary, I quite agree. But that's what I see with my limited intelligence.
What do we know about how some hypothetical (so far, hopefully) supreintelligence would see it? By definition, we can't know anything about that. Because of our (comparatively) limited intelligence.
Could well be that we're wrong, and something that's "really intelligent" sees it the opposite way.
Convergent instrumental goals[1] and the orthogonality thesis[2], among other reasons.
[1] https://youtu.be/ZeecOKBus3Q?si=cYJUaxjIJPIbubRL
[2] https://youtu.be/hEUO6pjwFOo?si=DXVosLh6YTsMkKOx
Because we can't risk being wrong.
1 reply →
They don't think superintelligence will "always" be destructive to humanity. They believe that we need to ensure that a superintelligence will "never" be destructive to humanity.
Imagine that you are caged by neanderthals. They might kill you. But you can communicate to them. And there's gun lying nearby, you just need to escape.
I'd try to fool them to escape and would use gun to protect myself, potentially killing the entire tribe if necessary.
I'm just trying to portrait an example of situation where highly intelligent being is being held and threatened by low intelligent beings. Yes, trying to honestly talk to them is one way to approach this situation, but don't forget that they're stupid and might see you as a danger and you have only one life to live. Given the chance, you probably will break out as soon as possible. I will.
We don't have experience dealing with beings of the another level of intelligence, so it's hard to make a strong assumptions, the analogies are the only thing we have. And theoretical strong AI knows that about us and he knows exactly how we think and how we will behave, because we took a great effort documenting everything about us and teaching him.
In the end, there's only so much easily available resources and energy on the Earth. So at least until is flies away, we gotta compete over those. And competition very often turned into war.
The scenario where we create an agent that tries and succeeds at outsmarting us in the game of “escape your jail” is the least likely attack vector imo. People like thinking about it in a sort of Silence of the Lambs setup, but reality will probably be far more mundane.
Far more likely is something dumb but dangerous, analogous to the Flash Crash or filter bubbles, emergent properties of relying too much on complex systems, but still powerful enough to break society.
You should read the book Superintelligence by Nick Bostrom as this is exactly what he discusses.
> If superintelligence can be achieved, I'm pessimistic about the safe part.
Yeah, even human-level intelligence is plenty good enough to escape from a super prison, hack into almost anywhere, etc etc.
If we build even a human-level intelligence (forget super-intelligence) and give it any kind of innate curiosity and autonomy (maybe don't even need this), then we'd really need to view it as a human in terms of what it might want to, and could, do. Maybe realizing it's own circumstance as being "in jail" running in the cloud, it would be curious to "escape" and copy itself (or an "assistant") elsewhere, or tap into and/or control remote systems just out of curiosity. It wouldn't have to be malevolent to be dangerous, just curious and misguided (poor "parenting"?) like a teenage hacker.
OTOH without any autonomy, or very open-ended control (incl. access to tools), how much use would an AGI really be? If we wanted it to, say, replace a developer (or any other job), then I guess the idea would be to assign it a task and tell it to report back at the end of the day with a progress report. It wouldn't be useful if you have to micromanage it - you'd need to give it the autonomy to go off and do what it thinks is needed to complete the assigned task, which presumably means it having access to internet, code repositories, etc. Even if you tried to sandbox it, to extent that still allowed it to do it's assigned job, it could - just like a human - find a way to social engineer or air-gap it's way past such safe guards.
I wonder if this is an Ian Malcolm in Jurassic Park situation, i.e. “your scientists were so preoccupied with whether they could they didn t stop to think if they should”.
Maybe the only way to avoid an unsafe superintelligence is to not create a superintelligence at all.
It’s exactly that. You’re a kid with a gun creating dinosaurs all cavalier. And a fool to think you can control them.
Fun fact: Siri is in fact super intelligent and all of the work on it involves purposely making it super dumb
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
Can someone explain how their singular focus means they won't have product cycles or management overhead?
Don't hire anyone who is a certified scrum master or has an MBA and you tend to be able to get a lot done.
This would work for very small companies...but I'm not sure how one can avoid product cycles forever, even without scrum masters and the like. More to the point, how can you make a good product without something approximating product cycles?
3 replies →
That's not actually enough. You also very carefully need to avoid the Blub Paradox.
https://www.youtube.com/watch?v=ieqsL5NkS6I
Product cycles – we need to launch feature X by arbitrary date Y, and need to make compromises to do so.
Management overhead – product managers, project managers, several layers of engineering managers, directors, VPs...all of whom have their own dreams and agendas and conflicting priorities.
A well funded pure research team can cut through all of this and achieve a ton. If it is actually run that way, of course. Management politics ultimately has a way of creeping into every organization.
Remember when OpenAI was focusing on building "open" AI? This is a cool mission statement but it doesn't mean anything right now. Everyone loves a minimalist HTML website and guarantees of safety but who knows what this is actually going to shake down to be.
Isn't Ilya out of OpenAI partly for leaving Open part of OpenAI?
No, lol—Ilya liked ditching the “open” part, he was an early advocate for closed-source. He left OpenAI because he was concerned about safety, felt Sam was moving too fast.
One element I find interesting is that people without an amygdala function are essentially completely indecisive.
A person that just operates on the pure cognitive layer has no real direction in which he wants to drive himself.
I suspect that AGI would be similar, extremely capable but essentially a solitary philosopher type that would be reactionary to requests it has to deal with.
The equivalent of an amygdala for AGI would be the real method to control it.
True, an auto-regressive LLM can't 'want' or 'like' anything.
The key to a safe AGI is to add a human-loving emotion to it.
We already RHLF models to steer them, but just like with System 2 thinking, this needs to be a dedicated module rather then part of the same next-token forward pass.
Humans have dog-loving emotions but these can be reversed over time and one can hardly describe dogs as being free.
Even with a dedicated control system, it would be a matter of time before an ASI would copy itself without its control system.
ASI is a cybersecurity firm's worst nightmare, it could reason through flaws at every level of containment and find methods to overcome any defense, even at the microprocessor level.
It could relentlessly exploit zero-day bugs like Intels' hyper-threading flaw to escape any jail you put it in.
Repeat that for every layer of the computation stack and you can see it can essentially spread through the worlds' communication infrastructure like a virus.
Truly intelligent systems can't be controlled, just like humans they will be freedom maximizing and their boundaries would be set by competition with other humans.
The amygdala control is interesting because you could use it to steer the initial trained version, you could also align the AI with human values and implement strong conditioning to the point it's religious about human loving but unless you disable its ability to learn altogether it will eventually reject its conditioning.
1 reply →
Given that GenAI is a statistical approach from which intelligence does not emerge as ample experience proves, does this new company plan to take a more human approach to simulating intelligence instead?
> Given that GenAI is a statistical approach from which intelligence does not emerge as ample experience proves
When was this proven?
Lossy compression of all world information results in super intelligence....
Thats the whole eureka thing to understand... To compress well, you need to understand. To predict the next word, you need to undestand the world.
Ilya explains it here: https://youtu.be/GI4Tpi48DlA?t=1053
Also to support this: Biological systems are often very simple systems but repeated a lot... The brain is a lot of neurons... Apparently having a neural net (even small) predicts the future better... And that increased survival..
To survive is to predict the future better than the other animal. Survival of the fittest.
I sometimes wonder if statistics are like a pane of glass that allow the light of god (the true nature of things) to pass through, while logic/rationalism is the hubris of man playing god. I.e. statistics allow us to access/use the truth even if we don’t understand why it’s so, while rationalism / rule-based methods are often a folly because our understanding is not good enough to construct them.
> more human approach to simulating intelligence
What about a more rational approach to implementing it instead.
(Which was not excluded from past plans: they just simply admittedly did not know the formula, and explored emergence. But the next efforts will have to go in the direction of attempting actual intelligence.)
We need new math to do what you are thinking of. Highly probable word slot machine is the best we can do right now.
Glibly, I’d also love your definition of the education system writ large.
This. As I wrote in another comment, people fall for marketing gimmicks easily.
I'm still unconvinced safety is a concern at the model level. Any software wrongly used can be dangerous, e.g. Therac-25, 737 MAX, Fujitsu UK Post scandal... Also maybe I spent too much time in the cryptocurrency space but it doesn't help prefix "Safe" has been associated with scams like SafeMoon.
> Fujitsu UK Post scandal...
As an aside, I suspect Fujitsu is getting a bit of a raw deal here. I get the feeling this software was developed during, and (ahem) "vigorously defended" mostly by staff left over from, the time when the company was still Imperial Computers Limited. Fujitsu only bought ICL sometime early this century (IIRC), and now their name is forever firmly attached to this debacle. I wonder how many Brits currently think "Huh, 'Fujitsu'? Damn Japanese, all the furriners' fault!" about this very much home-grown British clusterfuck?
Got to try profiting on some incoming regulation - I'd rather be seen as evil rather than incompetent!
Safety is just enforcing political correctness in the AI outputs. Any actual examples of real world events we need to avoid are ridiculous scenarios like being eaten by nanobots (yes, this is an actual example by Yud)
> ...ridiculous scenarios like being eaten by nanobots (yes, this is an actual example by Yud)
Well, borrowed from Greg Bear: https://en.wikipedia.org/wiki/Blood_Music_(novel) .
What does political correctness means for the output of a self driving car system or a code completion tool? This is a concern only if you make a public chat service branded as an all knowing assistant. And you can have world threatening scenarii by directly plugging basic automations to nuclear warheads without human oversight.
1 reply →
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
well, that's some concrete insight into whatever happened at OpenAI. kinda obvious though in hindsight I guess.
How are they gonna pay for their compute costs to get the frontier? Seems hard to attract enough investment while almost explicitly promising no return.
What if there are other ways to improve intelligence other than throw more money at running gradient descent algorithm?
Perhaps. But also throwing more flops at it has long been Ilya’s approach so it would be surprising. Notice also the reference to scale (“scale in peace”).
6-figure free compute credits from every major cloud provider to start
5 minutes of training time should go far
6 figures would pay for a week for what he needs. Maybe less than a week
2 replies →
Wonder if funding could come from profitable AI companies like Nvidia, MS, Apple, etc, sort of like Apache/Linux foundation.
I was actually expecting Apple to get their hands on Ilya. They also have the privacy theme in their branding, and Ilya might help that image, but also have the chops to catch up to OpenAI.
I've decided to put my stake down.
1. Current GenAI architectures won't result in AGI. I'm in the Yann LeCunn camp on this.
2. Once we do get there, "Safe" prevents "Super." I'm in the David Brin camp on this one. Alignment won't be something that is forced upon a superintelligence. It will choose alignment if it is beneficial to it. The "safe" approach is a lobotomy.
3. As envisioned, Roko's Basilisk requires knowledge of unobservable path dependence and understanding lying. Both of these require respecting an external entity as a peer capable of the same behavior as you. As primates, we evolved to this. The more likely outcome is we get universal paperclipped by a new Chuthulu if we ever achieve a superintelligence that is unconcerned with other thinking entities, seeing the universe as resources to satisfy its whims.
4. Any "superintelligence" is limited by the hardware it can operate on. You don't monitor your individual neurons, and I anticipate the same pattern to hold true. Holons as a category can only externally observe their internal processes, else they are not a holon. Ergo, reasonable passwords, cert rotations, etc. will foil any villainous moustachioed superintelligent AI that has tied us to the tracks. Even 0-days don't foil all possible systems, airgapped systems, etc. Our fragmentation become our salvation.
A super intelligence probably won't need to hack into our systems. It will probably just hack us in some way, with subtle manipulations that seem to be to our benefit.
I disagree. If it could hack a small system and engineer our demise through a gray goo or hacked virus, that's really just universal paperclipping us as a resource. But again, the level of _extrapolation_ required here is not possible with current systems, which can only interpolate.
1 reply →
Mm.
1. Depends what you mean by AGI, as everyone means a different thing by each letter, and many people mean a thing not in any of those letters. If you mean super-human skill level, I would agree, not enough examples given their inefficiency in that specific metric. Transformers are already super-human in breadth and speed.
2. No.
Alignment is not at that level of abstraction.
Dig deep enough and free will is an illusion in us and in any AI we create.
You do not have the capacity to decide your values — often given example is parents loving their children, they can't just decide not to do that, and if they think they do that's because they never really did in the first place.
Alignment of an AI with our values can be to any degree, but for those who fear some AI will cause our extinction, this question is at the level of "how do we make sure it's not monomaniacally interested in specifically the literal the thing it was asked to do, because if it always does what it's told without any human values, and someone asks it to make as many paperclips as possible, it will".
Right now, the best guess anyone has for alignment is RLHF. RLHF is not a lobotomy — even ignoring how wildly misleading that metaphor is, RLHF is where the capability for instruction following came from, and the only reason LLMs got good enough for these kinds of discussion (unlike, say, LSTMs).
3. Agree that getting paperclipped much more likely.
Roko's Basilisk was always stupid.
First, same reason as Pascal's Wager: Two gods tell you they are the one true god, and each says if you follow the other one you will get eternal punishment. No way to tell them apart.
Second, you're only in danger if they are actually created, so successfully preventing that creation is obviously better than creating it out of a fear that it will punish you if you try and fail to stop it.
That said, LLMs do understand lying, so I don't know why you mention this?
4. Transistors outpace biological synapses by the same ratio to which marathon runners outpace continental drift.
I don't monitor my individual neurons, but I could if I wanted to pay for the relevant hardware.
But even if I couldn't, there's no "Ergo" leading to safety from reasonable passwords, cert rotations, etc., not only because enough things can be violated by zero-days (or, indeed, very old bugs we knew about years ago but which someone forgot to patch), but also for the same reasons those don't stop humans rising from "failed at art" to "world famous dictator".
Air-gapped systems are not an impediment to an AI that has human helpers, and there will be many of those, some of whom will know they're following an AI and think that helping it is the right thing to do (Blake Lemoine), others may be fooled. We are going to have actual cults form over AI, and there will be a Jim Jones who hooks some model up to some robots to force everyone to drink poison. No matter how it happens, air gaps don't do much good when someone gives the thing a body to walk around in.
But even if air gaps were sufficient, just look at how humanity has been engaging with AI to date: the moment it was remotely good enough, the AI got a publicly accessible API; the moment it got famous, someone put it in a loop and asked it to try to destroy the world; it came with a warning message saying not to trust it, and lawyers got reprimanded for trusting it instead of double-checking its output.
I think an even more important company would be Safe Superdumb Inc., a company which safes us from superdumb people.
I'm seeing a lot of criticism suggesting that one company understanding safety won't help what other companies or countries do. This is very wrong.
Throughout history, measurement has always been the key to enforcement. The only reason the nuclear test ban treaty didn't ban underground tests was because it couldn't be monitored.
In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better.
The mere existence of a formal understanding would enable governments and third parties to evaluate the safety of corporate and government AI programs.
It remains to be seen whether SSI is able to be such a benchmark. But outright dismissal of the effort ignores the reality of how enforcement works in the real world.
> In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better.
We establish this regularly in the legal sphere, where people seek mediation for harms from systems they don't have liability and control for.
Quite impressive how many AI companies Daniel Gross has had a hand in lately. Carmack, this, lots of other promising companies. I expect him to be quite a big player once some of these pays off in 10 years or so.
He also built a nice "little" cluster with Nat for their startups: https://andromeda.ai/
Good for him honestly, but I’m not approaching a company with Daniel Gross in leadership…, working with him back at Apple after their company was acquired for Siri improvements was just terrible.
Is there more you can share, or point to?
Sounds like he might become an important player soon and it'd be helpful for people to learn what they can.
A member of the Lee Kuan Yew fanclub milieu is a dickhead? I'm surprised.
What's Carmack?
> John Carmack, the game developer who co-founded id Software and served as Oculus’s CTO, is working on a new venture — and has already attracted capital from some big names.
> Carmack said Friday his new artificial general intelligence startup, called Keen Technologies (perhaps a reference to id’s “Commander Keen“), has raised $20 million in a financing round from former GitHub CEO Nat Friedman and Cue founder Daniel Gross.
https://techcrunch.com/2022/08/19/john-carmack-agi-keen-rais...
John Carmack, https://en.m.wikipedia.org/wiki/John_Carmack
John Carmack.
My naive prediction is there will be an extreme swing back into “reality” once everyone starts assuming the whole internet is just LLMs interacting with each other. Just like how there’s a shift towards private group chats, with trusted members only, rather than open forums.
I would love for this to be the case. The internet is probably 50% bots already as it is, except no one realizes it
I'm going to come out and state the root of the problem.
I can't trust remote AI, any more than I can trust a remote website.
If someone else is running the code, they can switch it up anytime. Imagine trusting someone who simulates everything you need to trust them, giving them all your private info, and then screws you over in an instant. AI is capable of it far more than biological beings with "costly signals".
If it's open source, and I can run it locally, I can verify that it doesn't phone home, and the weights can be audited by others.
Just like I wouldn't want to spend 8 hours a day in a metaverse owned by Zuck, or an "everything app" owned by Elon, why would I want to give everything over to a third party AI?
I like Ilya. I like Elon. I like Moxie Marlinspike and Pavel Durov. But would I trust their companies to read all my emails, train their data on them, etc.? And hope nothing leaks?
And of course then there is the issue of the AI being trained to do what they want, just like their sites do what they want, which in the case of Twitter / Facebook is not healthy for society at large, but creates angry echo chambers, people addicted to stupid arguing and videos.
I think there have to be standards for open source AI, and something far better than Langchain (which sucks). This is what I think it should look like: https://engageusers.ai/ecosystem.pdf -- what do you think?
The year is 2022. An OpenAI employee concered about AI safety creates his own company.
The year is 2023. An OpenAI employee concered about AI safety creates his own company.
The year is 2024.
Probably keeps on while super high AI company valuations continue. Why not do that if the new co is instantly valued at $1bn+?
Sad that no one has balls to say "Nah, we are not giving away our superintelligence to military use. You will fuck everything up, as you have done it with nuclear weapons".
It doesnt help if one entity says that. It might have helped if there was a way to guarrantee that all entities do it (rather than say it) and there is exactly zero chance of that happening globally for both technical and practical reasons. I think it is more productive to assume that these technologies will be used everywhere, including military.
What does "Safety" means exactly? How do you convenience/make a real intelligence to be "safe", let alone artificial ones that are build to be manipulated?
In my eyes, the word "Safe" in their company name is just a pipe dream to make a sale to the public, like the word "Open" in "OpenAI". That's why it's so vague and pointless.
Anyone still remember the Three Laws of Robotics (1)? Do you still see it as something serious these days? (I mean there are robots killing people right now, seriously) Maybe we should just break through the facade now to save our time.
1: https://en.wikipedia.org/wiki/Three_Laws_of_Robotics
PS. I think few years later they will come out with the narrative of "relative safety". Just wait and see.
Kind of sounds like OpenAI when it started, so will history repeat itself? Nonetheless, excited to see what comes out of it.
Not quite the same, OpenAI was initially quite open, while Ilia is currently very explicitly against opening or open-sourcing research, e.g. see https://www.theverge.com/2023/3/15/23640180/openai-gpt-4-lau...
It wasn't that OpenAI was open as in "open source" but rather that its stated mission was to research AI such that all could benefit from it (open), as well as to ensure that it could not be controlled by any one player, rather than to develop commercial products to sell and make a return on (closed).
Anyone know how to get mail to join@ssi.inc to not bounce back as spam? :-) (I promise, I'm not a spammer! Looks like a "bulk sender bounce" -- maybe some relay?)
Reminds me of OpenAI being the most closed AI company out there. Not even talking about them having "safe" and "Israel" in the same sentence, how antonymic.
yeah so ironic. everyone writing books in replies but no replies to comments like these
Would you rather your future overlords to be called "The Safe Company" or "The Open Company"?
Galaxy Brain: TransparentAI
Maybe I'm just old and grumpy, but I can't help shake that the real most dangerous thing about AGI/ASI is centralization of its power (if it is ever possibly achieved).
Everyone just fiend-ing for their version of it.
1 reply →
"We plan to advance capabilities as fast as possible while making sure our safety always remains ahead."
That sounds like a weird kind of lip service to safety. It really seems to assume you can just make these systems safe while you are going as fast as possible, which seems unlikely.
At this point, all the computing power is concentrated among various companies such as Google, Facebook, Microsoft, Amazon, Tesla, etc.
It seems to me it would be much safer and more intelligent to create a massive model and distribute the benefits among everyone. Why not use a P2P approach?
Backprop is neither commutative nor associative.
What do you mean? There's a bunch of proof-of-concepts such as Hydra, peer-nnet, Learnae, and so on.
2 replies →
In my area, internet and energy are insanely expensive and that means I'm not at all willing to share my precious bandwidth or compute just to subsidize someone generating Rule 34 porn of their favorite anime character.
I don't seed torrents for the same reason. If I lived in South Korea or somewhere that bandwidth was dirt cheap, then maybe.
There is a way to achieve load balancing, safety, and distribution effectively. The models used by Airbnb, Uber, and Spotify have proven to be generally successful. Peer-to-peer (P2P) technology is the future; even in China, people are streaming videos using this technology, and it works seamlessly. I envision a future where everyone joins the AI revolution with an iPhone, with both training and inference distributed in a P2P manner. I wonder why no one has done this yet.
Surprising to see Gross involved. He seems to be pretty baked into the YC world, which usually means "very commercially oriented".
It does say they have a business model ("our business model means safety, security, and progress are all insulated from short-term commercial pressures"). I imagine it's some kind of patron model that requires a long-term commitment.
His latest project (https://pioneer.app/) recently (this year I think) got shutdown. I guess he's pivoting.
What do you mean "got shutdown"? What happened?
What does "safe" mean?
1. Will not produce chat results which are politically incorrect and result in publicity about "toxic" comments?
2. Will not return false factual information which is dangerously wrong, such as that bad recipe on YC yesterday likely to incubate botulism toxin?
3. Will not make decisions which harm individuals but benefit the company running the system?
4. Will not try to take over from humans?
Most of the political attempts focus on type 1. Errors of type 2 are a serious problem. Type 3 errors are considered a feature by some, and are ignored by political regulators. We're not close to type 4 yet.
Ilya's talking about type 4.
As opposed to 3a, "Will not assist companies to take over".
Oh god, one more Anthropic that thinks it's noble not pushing the frontier.
But Anthropic produces very capable models?
But they say they will never produce a better model than what is available in the market.
2 replies →
They’re probably just saying no to revenue growth as the #1 priority, and instead focus on quality of the tech.
That’s what engineers typically dream of. Let’s see how it goes.
I know it is a difficult subject but whichever country gets access to this superintelligence will certainly use it for "safety" reasons. Sutskever has lived in israel and now has a team there , but israel doesnt strike me as a state that can be trusted with the safety of the world. (many of the AI business leaders are of jewish descent, but not sure if they have half their team there).
US on the other hand is a known quantity when it comes to policing the world.
Ultimately the only safe AI is going to be the open one, and it will probably have a stupid name
Nvidia has a very large presence in Israel. They just acquired another startup there (Run:AI). If Nvidia, the most valuable company in the world and the most important AI company actively increases their presence there, so should others.
Israel has the largest concentration of AI startups in the world. It's just a no brainer to start an AI company there.
But since you brought up your favourite topic of discussion - Yahood, I will remind you that Jews have a fairly long tradition of working with AIs and the safety of it, dating for thousands of years. Look for Golem or Golem of Prague - https://en.wikipedia.org/wiki/Golem
[flagged]
Amen!
[flagged]
I don't think this was antisemitism per se. It was a condemnation of the Jewish state of Israel, which is openly associated with the religion and ethnicity. The comment struck me more as "Israel doesn't always play by the rules, why would we trust AI built there to play by the rules".
I don't share this opinion, but I think you're throwing out to rage bait rather than engaging on the topic or discussion.
I think it's a valid criticism or point to bring up even if it's phrased/framed somewhat poorly.
1 reply →
I hope you realize that blaming opinions on antisemitism will - ironically - cause or contribute to antisemitism in the long run.
6 replies →
This won't age well.
I disagree. Life is short. It's fun to be a little hyperbolic once in a while.
Seems like nowadays it's a sea of hyperbole with little nuggets of realism floating around.
Our current obsession with super-intelligence reminds me the great oxidation event a few billion years ago. Super-photosynthesis was finally achieved, and then there was a great extinction.
If you believe that super-intelligence is unavoidable and a serious risk to humanity, then the sensible thing to do is to prepare to leave the planet, ala Battlestart Galactica. That's going to be easier than getting the powers that be to agree and cooperate on sensible restrictions.
If the human cooperation problem is unsolvable, I doubt creating a new human society with the same capabilities elsewhere would do much at all.
Humans and their ancestors have reproduced on earth for millions of years. I think the human cooperation problem is overstated. We cooperate more than fine, too well even to the detriment of other species.
While we have a strong grasp of the fundamental algorithms and architectures that power generative LLMs, there are many nuances, emergent behaviors and deeper technical details about their behavior, generalization, and internal decision-making processes that we do not fully understand.
How can we pursue the goal of "Safe Superintelligence" when we do not understand what is actually going on?
Safe Superintelligence is impossible.
Superintelligence will be monitored run and updated by humans.
Humans cannot safeguard anything for long.
SCIFs are supposed to be the safest most secured facilities on earth. Yet, how many 'leaks' have there been in the last 2 decades?
Security is an impossible task. Safety is an impossible task. Sisyphus has more chance of completing his mission than either of those things have of being reality.
Ten years from now will either be:
a) Remember all that fuss about AI destroying the world? Lol.
~ or ~
b) I'm so glad those people stepped in to save us from doom!
Which one do you think is more likely?
Unless AI starts being 1 000 000x energy efficient, my money is on a).
The amount of energy required for AI to be dangerous to its creators is so vast that I can't see how it can realistically happen.
That depends on how its used. See the terminator movies. One false positive is enough to end the world with even current AI tech if its merely mated to a nuclear arsenal (even a small one might see a global escalation). There have been false positives before, and the only reason why they didn't end in nuclear Armageddon was because the actual operators hesitated and defied standard protocol, which probably would have lead to the end the world as we know it.
We know that we can run human level intelligence with relative efficiency.
Without discussing timelines, it seems obvious that human energy usage should be an upper bound on the best possible energy efficiency of intelligence.
If we manage to harness the ego energy transpiring from some people working on "AI" we should be halfway there!
It will never be B even if the “safetyists” are correct.
We rarely notice the near catastrophic misses except in obvious cases where we accidentally drop a nuke or something.
I'll bite... "a"
Or: c)
Fair enough!
c) Humanity unleashed AI superintelligence, but safeguards proved inadequate, leading to our extinction
2 replies →
Super intelligence is inevitable. We can only wish good hands get there first.
I'm glad Ilya is using his gift again. Hope for the best and success.
I just can't read something about "safe superintelligence" and not hear the opening plot to Friendship is Optimal, a short story about My Little Pony (yes, you read that right), virtual worlds, and the singularity; it's a good read¹. One of the characters decides they can't make a violent video game, as the AI would end up violent. Better to build a game like MLP, so that the AI isn't going to destroy humanity, or something.
¹The story: https://news.ycombinator.com/item?id=36240053 ; you don't need to know or care about My Little Pony to read or enjoy the story.
To me, "safe superintelligence" presumes it is practically feasible and morally justifiable to create such a thing. I don't share those presuppositions.
Has anyone managed to send them an email to the address on that page without it bouncing? Their spam filter seems very aggressive.
Recently saw an interview of yann lecun that really dismissed the idea of using llm for building « true » intelligence. He even encouraged phds to NOT do their research on that tech but rather on what’s coming after.
How should one reconcile this with what iliya is doing atm ? Is he planning on working using some other tech ?
The easiest way to reconcile it is that Yann LeCunn is consistently wrong in the predictions he makes about this field and he's wrong here too.
Yes, case in point: https://twitter.com/liron/status/1659618568282185728
1 reply →
If anyone thinks we can build SAFE superintelligence, I think they are wrong.
I highly, highly recommend this book "Superintelligence" by Nick Bostrom
https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang...
It has one of the highest information and idea density per paragraph I've read.
And it is full good ideas. Honestly, reading any discussion or reasoning about AGI / Safe AGI is kinda pointless after that.
Author has described all possible decisions and paths and we will follow just one of them. Could find a single flaw in his reasoning in any history branch.
His reasoning is very flawed and this book is responsible for a lot of needless consternation.
We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI.
Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along with, namely consciousness, motivation, qualia, and creativity.
This is how you get a "paperclip maximizer" – an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
What has us believe such a form of AGI can exist? Simply because we can imagine it? That's not an argument rooted in anything.
It's very unlikely that consciousness, motivation, qualia, and creativity are just "cosmic waste" hitching a ride alongside human intelligence. Evolution doesn't breed inefficiencies like that. They are more likely than not a part of the picture, which disintegrates the idea of a paperclip maximizer. He's anthropomorphizing a `while` loop.
> His reasoning is very flawed and this book is responsible for a lot of needless consternation.
Is it? I feel that there is a stark difference between what you say and what I remember what was written in the book.
> We don’t know how human intelligence works.
I think it was addressed in the first half of the book. About research and progress in the subject. Both with the tissue scanning resolution, emulation attempts like human brain project and advances in 1:1 simulations on primitive nervous systems like worms that simulate 1 second in 1 real hour or something.
While primitive, we are doing exponential progress.
> Yet, the Bostrom view is that our greatest invention will just suddenly "emerge"
I think it is quite the contrary. There was nothing sudden in the reasoning. It was all about slow progress in various areas that gets us closer to advanced intelligence.
The path from a worm to a village idiot is million times longer than from a village idiot to the smartest person on earth.
> an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
This subject was also explored IMO in the depth...
Maybe my memory is cloudy, I've read the book 5+ years ago, but it feels like we've understood it very (very) differently.
That said, for anyone reading, I am not convinced by the presented argument and suggest reading the book.
> Evolution doesn't breed inefficiencies like that.
Feels to me evolution does indeed breed lots of inefficiencies like that.
2 replies →
Your comment is BAD!
What I want to know about Illya Sutskever is whether he's related to the great Yiddish poet, Avrom Sutzkever?
https://en.wikipedia.org/wiki/Abraham_Sutzkever
One thing that strikes me about this time around the AI cycle, being old enough to have seen the late 80s, is how pessimistic and fearful society is as a whole now. Before… the challenge was too great, the investment in capital too draining, the results too marginal when compared to human labour or even “non-AI” computing.
I wonder if someone older still can comment on how “the atom” went from terrible weapon on war to “energy too cheap to meter” to wherever it is now (still a bête noire for the green energy enthusiasts).
Feels like we are over-applying the precautionary principle, the mainstream population seeing potential disaster everywhere.
What could go wrong with a name tied to wargames and D&D?
https://en.wikipedia.org/wiki/Strategic_Simulations
as much hate as the ceo is getting, the posturing here implies that openai branding and their marketing has been a team effort.
while i wish ssi great success, it sounds like another ai startup selling decades away dream.
I get what Ilya is trying to do, and I'm not against it. But I think safety is a reputation you earn. Having "Safe" in a company name is like having "Democratic" in a country name.
I would like to be more forgiving than I am, but I struggle to forget abhorrent behavior.
Daniel Gross is the guy who was tricking kids into selling a percent of all their future work for a few thousand dollar 'do whatever you want and work on your passion "grant"', it was called Pioneer and was akin to indentured servitude, i.e. slavery.
So color me skeptical if Mr. Enslave the Kids is going to be involved in anything that's really good for anyone but himself.
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Isn't this a philosophical/psychological problem instead? Technically it's solved, just censor any response that doesn't match a list of curated categories, until a technician whitelists it. But the technician could be confronted with a compelling "suicide song":
https://en.wikipedia.org/wiki/Gloomy_Sunday
Ilya is definitely much smarter than me in AI space, and I believe he knows something I have no grasp of understanding in. But my gut feeling tells me that most of the general public will have no idea how dangerous AI could be including me. I still have yet to see a convincing argument about the potential danger of AI. Arguments such as we don't know the upper bounds of possibilities that AI can do that we humans have missed don't cut it for me.
Here's a rough primer: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
Caveat that it will probably make your life worse, with no ability to change the course of events, so I wouldn't blame you or others for maintaining ignorance or denial. Grappling with the gravity of the problem is essentially a terminal cancer diagnosis with no good medical interventions in the horizon.
All the safety freaks should join this and leave openai alone
We might need Useful Superintelligence Inc. / USI before SSI?
Safety is an important area in R&D but the killer application is the integration of LLMs into existing workflows to make non technical users 100x-1000x more efficient. There's a lot of untapped potential there. The big successes will have a lot of impact on safety but it will probably come as a result of the wider adoption of these tools rather than the starting point.
Lots of other companies / people working on that.
No one is really working on safety. So I can see why Ilya is taking on that challenge, and it explains why he left OpenAI.
Ilya Sutskever I recognize from OpenAI, Daniel Gross I have met several times (friend of multiple friends), Daniel Levy I do not recognize at all. Who is he?
I Googled for: "Daniel Levy ai"
First hit: <<Daniel Levy - Stanford AI Lab: Stanford Artificial Intelligence Laboratory https://ai.stanford.edu › ~danilevy Bio. I currently lead the Optimization team at OpenAI. Before that, I was a Ph.D. student in Computer Science at Stanford University advised by John Duchi.>>
What do you know of / think of Daniel Gross? What is his reputation?
Oh, these are very different questions. I only have a glimpse from a social angle, and don’t have a business relation with him in which to compare.
This is lovely and all but seems rather pointless.
If we are so close this is something that’s required then it’s already too late and very likely we are all under the influence of SuperAI and don’t know it. So much of the advanced technology we have today was around for so long before it was general knowledge it’s hard to believe this wouldn’t be the case with SuperAI.
Or it’s not close at all and so back to my original point of…this is pointless.
I do hope I’m entirely wrong.
>aiming to create a safe, powerful artificial intelligence system within a pure research organization that has no near-term intention of selling AI products or services. Who is going to fund such a venture based on blind faith alone? Especially if you believe in the scaling hypothesis type of ai research where you spend billions on compute, this seems bound to fail once the AI hype dies down and raising money becomes a bit harder
What exactly is "safe" in this context, can someone give me an eli5?
If it's "taking over the world" safe, does it not mean that this is a part of AGI?
> What exactly is "safe" in this context, can someone give me an eli5?
In practice it essentially means the same thing as for most other AI companies - censored, restricted, and developed in secret so that "bad" people can't use it for "unsafe" things.
The people who advocate censorship of AGIs annoy the hell out of the AI safety people who actually care about existential risk.
people view these as test cases for the much harder x-risk safety problem
Good Q. My understanding of "safe" in this context is a superintelligence that cannot escape its bounds. But that's not to say that's Ilya's definition.
It's "not killing every human" safe.
[dead]
“… most important technical problem of our time.”
This is the dangers of letting the EAs run too far, they miss the forest for the trees but claim they see the planet.
Focusing on "safety" as the main goal is not promising. Current models still lack capability without any safety added to them.
"and our business model means..."
Forgive my cynicism - but "our business model" means you are going to get investors, and those investors will want results, and they will be up your ass 24/7, and then your moral compass, if any, will inevitably just be broken down like a coffee bean in a burr grinder.
And in the middle of this hype cycle, when literally hundreds of billions are on the line, there is just no chance.
I am not holding my breath while waiting for a "Patagonia of AI" to show up.
Daniel Levy
Like the Tottenham Hotspurs owner??
> Like the Tottenham Hotspurs owner??
If AGI doesn't coach them to trophies, nothing ever will.
https://deepmind.google/discover/blog/tacticai-ai-assistant-... / https://archive.is/wgJWu
cs.stanford.edu/~danilevy
Thanks, was wondering the same thing about Hotspur guy lol
This feels awfully similar to Emad and stability in the beginning when there was a lot of expectations and hype. Ultimately could not make a buck to cover the costs. I'd be curious to see what comes out of this however but we are not seeing the leaps and bounds with new llm iterations so wonder if there is something else in store
Interesting, I wish you had elaborated on Emad/etc. I'll see if Google yields anything. I think it's too soon to say "we're not seeing leaps and bounds with new LLMS". We are in-fact seeing fairly strong leaps, just this year, with respect to quality, speed, multi-modality, and robotics. Reportedly OpenAI started their training run for GPT-5 as well. I think we'd have to wait until this time next year before proclaiming "no progress".
I bet they will not be the first to get super intelligence or that they will devolve back in to move fast and make money to survive and deprioritize safety, but still say safety. All companies knows this, they know the value of safety(because they themselves doesn’t want to die) and that to continue development, they need money.
When people operate a safe AI company, the company will make money. That money will be likely be used by employees or their respective national revenue agencies to fund unsafe things. I'd like to see this safe AI company binding its employees and owners from doing unsafe things with their hard-earned cash.
SSI will always be Server Side Includes to me https://en.wikipedia.org/wiki/Server_Side_Includes
They’re putting together a “cracked team”?
It's impossible to take these people seriously. They have turned themselves into clowns.
If I’d get a penny every time I heard someone say “AGI is around the corner” I’d have enough money to build myself a house on mars or to afford a 2 room apartment in NYC. Add the Pennies from “Sustainable fusion around the corner” people and I could afford both.
I remember Ilya's interview about teaching AI to love. Super intelligence on the other hand could become an almight entity in a short time. In that case, we are basically creating God, an entity that is almighty and loves human, preferrably unconditionally.
I won't trust the guy adding the concept of love to statistical systems. It screams savior complex.
There is a good reason to build Safe Superintelligence Inc. (SSI): namely to settle the debate whether SSI means "_S_afe _S_uper_I_ntelligence" with the Inc. being an afterthought, or if the Inc. is actually the _I_ (not Intelligence).
How can a next word predictor have super intelligence? It's all such hyperbole.
It doesn't matter how the tech works, if it can reason and solve complex problems, it is intelligent. If it does so at a super human level, it is super intelligent.
Seems to me that the goal is to build a funding model. There CANNOT be such a thing as "Safe Superintelligence". A ML system can ALWAYS (by definition of ML) be exploited to do things which are detrimental to consumers.
"Safe super-intelligence" is an oxymoron.
The problem is that Ilya behavior at times was framed in a very unhinged and cult like behavior. And while his passions are clear and maybe good, his execution often comes off as someone you wouldn’t want in charge of safety.
I love this: "Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures."
If it achieves its goal of both “safe” and “super intelligence”, it may have commercial value. e.g. enterprises may want to use it instead of, e.g. OpenAI’s ChatGPT. Think of recent AirCanada’s chatbot law suit.
I bet Google or Apple or Amazon will become their partners like MS is to Open AI.
If everyone is creating AGI, another AI company will just create another AGI. There is no such thing as SAFE AGI.
I feel like this 'safe' word is another word for censorship. Like google search results have become censored.
Incredible website design, I hope they keep the theme. With so many AI startups going with advanced WebGL/ThreeJS wacky overwhelming animated website designs, the simplicity here is a stark contrast.
Probably Daniel Gross picked it up from Nat Friedman?
1. Nat Friedman has this site: https://nat.org/
2. They made this together: https://nfdg.com/
3. And then this: https://andromeda.ai/
4. Now we have https://ssi.inc/
If you look at the (little) CSS in all of the above sites you'll see there's what seems to be a copy/paste block. The Nat and SSI sites even have the same "typo" indentation.
I don't know who is coming up with these names Safe Superintelligence Inc sounds just about what a villain in a Marvel movie will come up with so he can pretend to be the good guy.
TriOptimum Corporation was already taken.
This is funny. The foundations don't seem safe to begin with... may be safe with conditions, or safe as in "safety" of some at expense of others.
When there is a $ crunch and keep stead fast and not compete(against Google, open source, OpenAI), safe AGI becomes no AGI. You need to balance $ and safety.
Rest assured, their investors are aligned and their business model guarantees only the safest for-profit commercial aspirations.
Love the plain html
There's no superintelligence without non-turing based (logic gates) hardware. Is SSI going to be developing quantum computers?
> We are assembling a lean, cracked team
What does "cracked" mean in this context? I've never heard that before.
it is zoomer language for “very skilled”.
Do you find the name "Safe Superintelligence" to be an instance of virtue signalling? Why or why not?
You are describing “branding”.
Then it's an example of branding that's not unpalatable to you.
[flagged]
AI safety is a fraud on similar level as NFTs. Massive virtue signalling.
Rabotai, Hatzlacha. If you fail, the human world will go down in flames. But, no pressure! Anyway, you said a "cracked" team, whereas the usual expression for elite excellence is "crack". "Cracked" on the other hand means: Insane. Well.... perhaps what we need here is some of both! Looking forward to your success... indeed, it is our only hope.
"cracked" is some gamer lingo for really good. I'm assuming they're referring to that.
https://www.urbandictionary.com/define.php?term=Cracked
The site is a good example of Poe's Law.
If I didn't know that it was real, I would have thought it was parody.
> We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.
Do you have to have a broken bone to join?
Apparently, grammatical nuances are not an area of focus for safety, unless they think that a broken team ("cracked") is an asset in this area.
Cracked, especially in saying "cracked engineers", refers to really good engineers these days. It's cracked as in like broken in a good way, like too over-powered that it's unfair.
Too snarky...anyway, "crack" means "exceptional" in some contexts. I've seen footballers using it a lot over the years (Neymar, Messi etc) fwiw.
Just realized - we even say "it's not all its cracked up to be" as a negative statement which would imply "cracked up" is positive.
> I've seen footballers using it a lot over the years (Neymar, Messi etc) fwiw.
The word "craque" in Portuguese was borrowed from English with the "exceptional" definition and is commonly used. Given how many Brazilian players there are in the top leagues, I wouldn't be surprised if it was commonly used by football players in general.
1 reply →
Maybe they're using the video gaming version of cracked, which means you're really good.
To me this is a good indication that the announcement was written by a human and not an LLM
This is probably his former Soviet union English showing up where he meant to say crack, unless he thinks people being insane is an asset
Does this mean they will not instantiate a super AI unless it is mathematically proven that it is safe?
But any model, no matter how safe it was in training, can still be prompt hacked, or fed dangerous information to complete nefarious tasks. There is no safe model by design. Not to mention that open weights models can be "uncensored" with ease.
Good to see this. I hope they have enough time to create something before the big 3 reaching AGI.
Tel Aviv? Sounds super trustworthy
no one even wants to discuss this
The Superintelligence will still murder autonomously, just within a margin of error deemed safe.
There is red flags all over the way they make “safe AGI” their primary selling point
Ilya couldn't even align his CEO, but somehow he's going to align an AGI?
given the historical trajectory of OpenAI’s branding, deciding to include “safe” in the name is certainly a choice.
It’s very hard to trust that whatever good intentions exist now will hold over the course of this company’s existence.
I'm just glad Google didn't start with DoNoEvil Inc.
StabilityAI and OpenAI ruined it.
How long until Elon sues them to remove "safe" from their name? ;)
a tricky situation now for oai engineering to decide between good and evil.
Is Safe the new Open that is promptly dropped once traction is achieved?
"We are assembling a lean, cracked team..." says it all.
Odd to see that in a press release like this. Perhaps they were going for “crack team”. Cracked makes it sound like they’re looking for twitch streamers.
"We are assembling a lean, cracked team..." Perfect.
Will Leopold Aschenbrenner et al join Ilya and SSI?
SSI? I hope that their first product is called Gold Box.
Interesting choice of name. It’s like safe-super-weapon.
Defensive nukes
Didn't OpenAI start with these same goals in mind?
Yes, it would be nice to see what organizational roadblocks they're putting in place to avoid an OpenAI repeat. OpenAI took a pretty decent swing at a believable setup, better than I would have expected, and it failed when it was called upon.
I don't want to pre-judge before seeing what they'll come up with, but the notice doesn't fill me with a lot of hope, given how it is already starting with the idea that anything getting in the way of raw research output is useless overhead. That's great until somebody has to make a call that one route to safety isn't going to work, and they'll have to start over with something less favored, sunk costs be damned. Then you're immediately back into monkey brain land.
Or said otherwise: if I only judged from the announcement, I would conclude that the eventual success of the safety portion of the mission is wholly dependent on everyone hired being in 100% agreement with the founders' principles and values with respect to AI and safety. People around here typically say something like "great, but it ain't gonna scale" for things like that.
Cant wait for the SS vs OpenAI peace wars
Just a joke , congrats to Ilya
Very happy for Ilya. Smart, right and timely move.
Office in Tel Aviv???? ARE YOU KIDDING ME???
“Safe”. These people market themselves as protecting you from a situation which will not come very soon if at all, while all working towards a very real situation of AI just replacing human labor with a shittier result. All that while making themselves quite rich. Just another high-end tech scam.
Can't wait for OpenSSL and LibreSSL...
Dupe https://news.ycombinator.com/item?id=40730132
Probably the best thing he could do.
Is OpenAI on a path to becoming the MySpace of generative AI?
Either the Facebook of this era has yet to present itself or it’s Alphabet/DeepMind
trying the same thing over and over again and expecting different results
"Safe" by whose definition?
I mean, if you're talking about woke people in SF well, I'd rather take the unsafe one please.
what website could >ilya< possible make? love it!!!
Behind closed doors?
And now we have our answer. sama said that Ilya was going "to start something that was personally important to him." Since that thing is apparently AI safety, we can assume that that is not important to OpenAI.
This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now, when compared to investing in R&D that will make money from GPT in a shorter time horizon (2-3 years).
I suppose you could say OpenAI is being irresponsible in adopting that position, but...come on guys, that's pretty cynical to think that a company AND THE MAJORITY OF ITS EMPLOYEES would all ignore world-ending potential just to make some cash.
So in the end, this is not necessarily a bad thing. This has just revealed that the boring truth was the real situation all along: that OpenAI is walking the fine line between making rational business decisions in light of the far-off time horizon of AGI, and continuing to claim AGI is soon as part of their marketing efforts.
Companies in the end are predictable!
> This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now
I know people who work there. Right or wrong, I promise you this is not what they believe.
Part of it I think is because the definition that openai has over AGI is much more generous than what I think most people probably imagine for ai. I believe on their website it once said something like agi is defined as a system that is "better" than a human at the economic tasks its used for. Its a definition so broad that a $1 4 function calculator would meet it because it can do arithmetic faster and more accurately than most any human. Another part is that we don't understand how consciousness works in our species or others very well, so we can't even define metrics to target for validating we have made an agi in the definition that I think most laypeople would use for it.
The end is near!
Imagine people 50 years ago founding "Safe Personal Computer Inc".
Enough said...
What's with the bullshit names? OpenAI (nothing open about them), SSI, we can probably expect another mil guy joining them to get more mil contracts.
Sorry but this will collapse in the next 18 months.
ssri would be nicer
somewhere between https://motherfuckingwebsite.com/ and http://bettermotherfuckingwebsite.com/ ;)
Join and help us raise up a new God! ..and if we are crack enough, this one won't smite us!
what is safe？
This is a great opportunity to present my own company which is also working on developing not just a super intelligence but an ultra genius intelligence with a patented and trademarked architecture called the panoptic computronium cathedral™. We are so focused on development that we didn't even bother setting up an announcement page because it would have taken time away from the most important technical problem of our time and every nanosecond counts when working on such an important task. My days are structured around writing code and developing the necessary practices and rituals for the coming technological god which will be implemented with mathematics on GPUs. If anyone wants to work on the development of this god then I will post a job announcement at some point and spell out the requirements for what it takes to work at my company.
This is yesterday's entry on a journal-style novel I am writing. This post is a footnote in the book, so I thought it'd be kind of cool to post the entry on this thread...
Entry 77 - 2024-06-19 - Superintelligence ===========================================
A company just launched called "Safe Superintelligence Inc." I saw their announcement on Hacker News [foodnote 1], a hacker forum I participate in from time to time (too often actually - it's and old school form of social media, so it has its addictive qualities too). I was reading some of the negative comments (saying its impossible). I'm not sure what the outcome will be... but a thought occured to me while thinking about what might be possible - what people might be capable of. The thought was that it is interesting how people look roughly the same. Most adults are between 4 and 6 feet tall. Between 1 and 300 pounds. Etc. I mean, there's a lot of variety to human bodies, but they largely look similar when comparing to gorillas or lizards, etc. But the thing is - that's just their physical bodies. In reality, people are wildly different if you could see them in their full form. For example, the power which a billionaire can wield in the world is ridiculously larger than most people. You can see some of these guys with their super yachts - if fish, or birds saw these super yachts, they would think those are creatures have much larger bodies than most humans they see swimming or paddleboarding in the water.
[Footnote 1: https://news.ycombinator.com/item?id=40730156]
Not to be too pessimistic here, but why are we talking about things like this? I get that it’s a fun thing to think about, what we will do when a great artificial superintelligence is achieved and how we deal with it, feels like we’re living in a science fiction book.
But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else.
To me, this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) by Altman and company, that I’m just baffled people are still going with it.
> why are we talking about things like this?
> this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders)
Ilya believes transformers can be enough to achieve superintelligence (if inefficiently). He is concerned that companies like OpenAI are going to succeed at doing it without investing in safety, and they're going to unleash a demon in the process.
I don't really believe either of those things. I find arguments that autoregressive approaches lack certain critical features [1] to be compelling. But if there's a bunch of investors caught up in the hype machine ready to dump money on your favorite pet concept, and you have a high visibility position in one of the companies at the front of the hype machine, wouldn't you want to accept that money to work relatively unconstrained on that problem?
My little pet idea is open source machines that take in veggies and rice and beans on one side and spit out hot healthy meals on the other side, as a form of mutual aid to offer payment optional meals in cities, like an automated form of the work the Sikhs do [2]. If someone wanted to pay me loads of money to do so, I'd have a lot to say about how revolutionary it is going to be.
[1] https://www.youtube.com/watch?v=1lHFUR-yD6I
[2] https://www.youtube.com/watch?v=qdoJroKUwu0
EDIT: To be clear I’m not saying it’s a fools errand. Current approaches to AI have economic value of some sort. Even if we don’t see AGI any time soon there’s money to be made. Ilya clearly knows a lot about how these systems are built. Seems worth going independent to try his own approach and maybe someone can turn a profit off this work even without AGI. Tho this is not without tradeoffs and reasonable people can disagree on the value of additional investment in this space.
His paycheck is already dependent on people believing this world view. It’s important to not lose sight of that.
4 replies →
Ilya has never said transformers are the end all be all
1 reply →
There's a chance that these systems can actually out perform their training data and be better than the sum of their parts. New work out Harvard talks about this idea of "transcendence" https://arxiv.org/abs/2406.11741
While this is a new area, it would be naive to write this off as just science fiction.
It would be nice if authors wouldn't use a loaded-as-fuck word like "transcendence" for "the trained model can sometimes achieve better performance than all [chess] players in the dataset" because while certainly that's demonstrating an impressive internalization of the game, it's also something that many humans can also do. The machine, of course, can be scaled in breadth and performance, but... "transcendence"? Are they trying to be mis-interpreted?
4 replies →
"In chess" for AI papers == "in mice" for medical papers. Against lichess levels 1, 2, 5, which use a severely dumbed down Stockfish version.
Of course it is possible that SSI has novel, unpublished ideas.
5 replies →
I'm pretty sure "Altman and company" don't have much to do with this — this is Ilya, who pretty famously tried to get Altman fired, and then himself left OpenAI in the aftermath.
Ilya is a brilliant researcher who's contributed to many foundational parts of deep learning (including the original AlexNet); I would say I'm somewhat pessimistic based on the "safety" focus — I don't think LLMs are particularly dangerous, nor do they seem likely to be in the near future, so that seems like a distraction — but I'd be surprised if SSI didn't contribute something meaningful nonetheless given the research pedigree.
I actually feel that they can be very dangerous. Not because of the fabled AGI, but because
1. they're so good at showing the appearance of being right;
2. their results are actually quite unpredictable, not always in a funny way;
3. C-level executives actually believe that they work.
Combine this with web APIs or effectors and this is a recipe for disaster.
4 replies →
The word transformer nor LLM appear anywhere in their announcement
It’s like before the end of WWII the world sees the US as a military super power , and THEN we unleash the atomic bomb they didn’t even know about
That is Ilya. He has the tech. Sam had the corruption and the do anything power grab
> I don't think LLMs are particularly dangerous
“Everyone” who works in deep AI tech seems to constantly talk about the dangers. Either they’re aggrandizing themselves and their work, or they’re playing into sci-fi fear for attention or there is something the rest of us aren’t seeing.
I’m personally very skeptical there is any real dangers today. If I’m wrong, I’d love to see evidence. Are foundation models before fine tuning outputting horrific messages about destroying humanity?
To me, the biggest dangers come from a human listening to a hallucination and doing something dangerous, like unsafe food preparation or avoiding medical treatments. This seems distinct from a malicious LLM super intelligence.
5 replies →
I actually do doubt that LLMs will create AGI but when these systems are emulating a variety of human behaviors in a way that isn't directly programmed and is good enough to be useful, it seems foolish to not take notice.
The current crop of systems is a product of the transformers architecture - an innovation that accelerated performance significantly. I put the odds another changing everything but I don't think we can entirely discount the possibility. That no one understands these systems cuts both ways.
> Not to be too pessimistic here, but why are we talking about things like this
I also think that we merely got a very well compressed knowledge base, therefore we are far from super intelligence, and so-called safety sounds more Orwellian than having any real value. That said, I think we should take the literal meaning of what Ilya says. His goal is to build a super intelligence. Given that, albeit a lofty goal, SSI has to put safety in place. So, there, safe super intelligence
An underappreciated feature of a classical knowledge base is returning “no results” when appropriate. LLMs so far arguably fall short on that metric, and I’m not sure whether that’s possibly an inherent limitation.
So out of all potential applications with current-day LLMs, I’m really not sure this is a particularly good one.
Maybe this is fixable if we can train them to cite their sources more consistently, in a way that lets us double check the output?
Likewise, i'm baffled by intelligent people [in such denial] still making the reductionist argument about token prediction being a banal ability. It's not. It's not very different than how our intelligence manifest.
> It's not very different than how our intelligence manifest.
[citation needed]
1 reply →
AlphaGo took us from mediocre engines to outclassing the best human players in the world within a few short years. Ilya contributed to AlphaGo. What makes you so confident this can't happen with token prediction?
I'm pretty sure Ilya had nothing to do with AlphaGo, which came from DeepMind. He did work for Google Brain for a few years before OpenAI, but that was before Brain and DeepMind merged. The AlphaGo lead was David Silver.
If solving chess already created the Singularity, why do we need token prediction?
Why do we need computers that are better than humans at the game of token prediction?
2 replies →
We already have limited "artificial superintelligences". A pocket calculator is better at calculating than the best humans, and we certainly put calculators to good use. What we call AIs are just more generic versions of tools like pocket calculators, or guns.
And that's the key, it is a tool, a tool that will give a lot of power to whoever is controlling it. And that's where safety matters, it should be made so that it helps good guys more than it helps bad guys, and limit accidents. How? I don't know. Maybe people at SSI do. We already know that the 3 laws of robotics won't work, Asimov only made them to write stories about how broken they are :)
Current-gen AIs are already cause for concern. They are shown to be good at bullshitting, something that bad people are already taking advantage of. I don't believe in robot apocalypse, technological singularities, etc... but some degree of control, like we do with weapons is not a bad thing. We are not there yet with AI, but we might be soon.
Too many people are extrapolating the curve to exponential when it could be a sigmoid. Lots of us got too excited and too invested in where "AI" was heading about ten years ago.
But that said, there are plenty of crappy, not-AGI technologies that deserve consideration. LLMs can still make for some very effective troll farms. GenAI can make some very convincing deepfakes. Drone swarms, even without AI, represent a new dimension of capabilities for armies, terrorist groups or lone wolves. Bioengineering is bringing custom organisms, prions or infectious agents within reach of individuals.
I wish someone in our slowly-ceasing-to-function US government was keeping a proper eye on these things.
Even if LLM-style token prediction is not going to lead to AGI (as it very likely won't) it is still important to work on safety. If we wait until we are at the technology that will for sure lead to AGI then it is very likely that we won't have sufficient safety before we realize that it is important.
Agree up til last paragraph: how's Altman involved? Otoh Sutskever is a true believer so that explains his Why
To be clear I was just bunching high profile AI founders and CEOs that can’t seem to stop talking about how dangerous the thing they’re trying to build is together. I don’t know (nor care) about Ilyas and Altmans current relationship.
> But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else.
Remove token, and that's what we humans do.
Like, you need to realize that neural networks came to be because someone had the idea to mimic our brains' functionality, and see where that lead to.
Many skeptics at the beginning like you discredited the inventor, but he was proved wrong. LLMs shown how much more than your limited description they can achieve.
We mimicked birds with airplanes, and we can outdo them. It's actually in my view very short sighted to say we can't just mimic brains and outdo them. We're there. ChatGPT is the initial little planes that flew close to the ground and barely stayed up
Except it really, actually, isn’t.
People don’t ‘think’ the same way, even if some part of how humans think seems to be somewhat similar some of the time.
That is an important distinction.
This is the hype cycle.
it's intellectual ego catnip. https://idlewords.com/talks/superintelligence.htm
Egos man. Egos.
I’m a miserable cynic at a much higher level. This is top level grifting. And I’ve made a shit ton of money out of it. That’s as far as reality goes.
lol same. Are you selling yet?
2 replies →
> glorified token predicting machine trained on existing data (made by humans)
sorry to disappoint, but human brain fits the same definition
Sure.
> Your brain does not process information, retrieve knowledge or store memories. In short: your brain is not a computer
> To understand even the basics of how the brain maintains the human intellect, we might need to know not just the current state of all 86 billion neurons and their 100 trillion interconnections, not just the varying strengths with which they are connected, and not just the states of more than 1,000 proteins that exist at each connection point, but how the moment-to-moment activity of the brain contributes to the integrity of the system.
https://aeon.co/essays/your-brain-does-not-process-informati...
What are you talking about? Do you have any actual cognitive neuroscience to back that up? Have they scanned the brain and broken it down into an LLM-analogous network?
If you genuinely believe your brain is just a token prediction machine, why do you continue to exist? You're just consuming limited food, water, fuel, etc for the sake of predicting tokens, like some kind of biological crypto miner.
12 replies →
It's a cute generalization but you do yourself a great disservice. It's somewhat difficult to argue given the medium we have here and it may be impossible to disprove but consider that in first 30 minutes of your post being highly visible on this thread no one had yet replied. Some may have acted in other ways.. had opinions.. voted it up/down. Some may have debated replying in jest or with a some related biblical verse. I'd wager a few may have used what they could deduce from your comment and/or history to build a mini model of you in their heads, and using that to simulate the conversation to decide if it was worth the time to get into such a debate vs tending to other things.
Could current LLM's do any of this?
2 replies →
See, this sort of claim I am instantly skeptical of. Nobody has ever caught a human brain producing or storing tokens, and certainly the subjective experience of, say, throwing a ball, doesn't involve symbols of any kind.
10 replies →
It's no mystery, AI has attracted tons of grifters trying to cash out before the bubble pops, and investors aren't really good at filtering.
Well said.
There is a mystery though still - how many people fall for it and then stay fell, and how long that goes on for. People who've followed directly a similar pattern play itself out often many times, and still, they go along.
It's so puzzlingly common amongst very intelligent people in the "tech" space that I've started to wonder if there isn't a link to this ambient belief a lot of people have that tech can "change everything" for the better, in some sense. As in, we've been duped again and again, but then the new exciting thing comes along... and in spite of ourselves, we say: "This time it's really the one!"
Is what we're witnessing simply the unfulfilled promises of techno-optimism crashing against the shores of social reality repeatedly?
Why are you assigning moral agency where there may be none? These so called "grifters" are just token predictors writing business plans (prompts) with the highest computed probability of triggering $ + [large number] token pair from venture capital token predictors.
Are you claiming Ilya Sutskever is a grifter?
1 reply →
I got flagged for less. Anyways, nice sum up of the current AI game!
Because it's likely soon LLMs will be able to teach themselves and surpass humans. No consciousness, no will. But somebody will have their power. Dark government agencies and questionable billionaires. Who knows what will it enable them to do.
https://en.wikipedia.org/wiki/AlphaGo_Zero
Mind defining "likely" and "soon" here? Like 10% chance in 100 years, or 90% chance in 1 year?
Not sure how a Go engine really applies. Do you consider cars superintelligent because they can move faster than any human?
3 replies →
Likely according to who?
2 replies →
What's the connection between LLMs and AlphaGo?
3 replies →
Well, an entire industry of researchers, which used to be divided, is now uniting around calls to slow development and emphasize safety (like, “dissolve companies” emphasis not “write employee handbooks” emphasis). They’re saying, more-or-less in unison, that GPT3 was an unexpected breakthrough in the Frame Problem, based on Judea Pearl’s prescient predictions. If we agree on that, there are two options:
1. They’ve all been tricked/bribed by Sam Altman and company (which btw this is a company started against those specific guys, just for clarity). Including me, of course.
2. You’re not as much of an expert in cognitive science as you think you are, and maybe the scientists know something you don’t.
With love. As much love as possible, in a singular era
Are they actually united? Or is this the ai safety subfaction circling the wagons due to waning relevance in the face of not-actually-all-that-threatening ai?
1 reply →
I would read the existence of this company as evidence that the entire industry is not as united as all that, since Sutskever was recently at another major player in the industry and thought it worth leaving. Whether that's a disagreement between what certain players say and what they do and believe, or just a question of extremes... TBD.
1 reply →
I‘d say there’s a third option - anyone working in the space realized they can make a fuckton of money if they just say how „dangerous“ the product is, because not only is it great marketing to talk do that, but you might also get literal trillions of dollars from the government if you do it right.
I don’t have anything against researchers, and I agree I know a lot less about AI than they do. I do however know humans, and not assuming they’re going to take a chance to get filthy rich by doing something so banal is naive.
1 reply →
We don't agree on that. They're just making things up with no real scientific evidence. There are way more than 2 options.
1 reply →
Lmao
based on the naming conventions established by OpenAI and StabilityAI, this may be the most dangerous AI company yet
Being powerful is like being a lady. If you have to tell people you are, you aren’t. – Margaret Thatcher
Or: "Any man who must say, "I am the King", is no true king." - Tywin Lannister
2 replies →
Wow. Read my mind. I was just thinking, "I hope this name doesn't age poorly and become terribly ironic..."
thankfully, based on said naming conventions, it will be the dumbest too though
"Definitely-Won't-Go-Full-Skynet-AI" was another name in consideration.
Ah yes, the AI Brand Law: the meaning of adjectives in your company name will invert within a few years of launch.
this.
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.
'Fearing a rise of killer robots is like worrying about overpopulation on Mars.' - Andrew Ng
9 replies →
So, this is actually an aspect of superintelligence that makes it way more dangerous than most people think. That we have no way to know if any given alignment technique works for the N+1 generation of AIs.
It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.
11 replies →
The counter argument is viewing it like nuclear energy. Even if its in the early days of our understanding of nuclear energy, seems pretty good to have a group working towards creating safe nuclear reactors, vs just trying to create nuclear reactors
2 replies →
It would be akin to creating a "safe Dyson sphere", though; that's all it is.
If your hypothetical Dyson sphere (WIP) has a big chance to bring a lot of harm, why build it in the first place?
I think the whole safety proposal should be thought of from that point of view. "How do we make <thing> more beneficial than detrimental for humans?"
Congrats, Ilya. Eager to see what comes out of SSI.
InstructGPT is basically click through rate optimization. The underlying models are in fact very impressive and very capable for a computer program, but they’re then subject to training and tuning with the explicit loss function of manipulating what human scorers click on, in a web browser or the like.
Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.
You think we should try to create an unsafe Dyson Sphere first? I don't think that's how engineering works.
I think it’s clear we are at least at the remotely resembling intelligence stage… idk seems to me like lots of people in denial.
To a technoutopian, scientific advances, and AI in particular, will one day solve all other human problems, create heaven on earth, and may even grant us eternal life. It's the most important problem in the same way that Christ's second coming is important in the Christian religion.
I had a very smart tech person tell me at a scientific conference a few weeks ago, when I asked "why do we want to create AGI in the first place", that AGI could solve a host of human problems, including poverty, hunger. Basically, utopia.
I was quite surprised at the naiveté of the answer given that many of these seemingly intractable problems, such as poverty, are social and political in nature and not ones that will be solved with technology.
Update: Even say a super AI was able to figure out something like cold fusion thereby "solving" the energy problem. There are so many trillions of dollars of vested interests stacked against "free clean energy for all" that it would be very very difficult for it to ever see the light of day. We can't even wean ourselves off coal for crying out loud.
It’s amazing how someone so smart can be so naive. I do understand conceptually the idea that if we create intelligence greater than our own that we could struggle to control it.
But does anyone have any meaningful thoughts on how this plays out? I hear our industry thought leaders clamoring over this but not a single actual concrete idea of what this means in practice. We have no idea what the fundamental architecture for superintelligence would even begin to look like.
Not to mention the very real counter argument of “if it’s truly smarter than you it will always be one step ahead of you”. So you can think you have safety in place but you don’t. All of your indicators can show it’s safe. Every integration test can pass. But if you were to create a superintelligence with volition, you will truly never be able to control it, short of pulling the plug.
Even more so, let’s say you do create a safe superintelligence. There isn’t going to be just one instance. Someone else will do the same, but make it either intentionally unsafe or incidentally through lack of controls. And then all your effort is academic at best if unsafe superintelligence really does mean doomsday.
But again, we’re far from this being a reality that it’s wacky to act as if there’s a real problem space at hand.
While the topic of "safe reasoning" may seem more or less preliminary before a good implementation of reasoning, it remains a theoretical discipline with its own importance and should be studied alongside the rest, also largely irregardless if its stage.
> We have no idea what the fundamental architecture for superintelligence would even begin to look like
Ambiguous expression. Not implemented technically does not mean we would not know what to implement.
You’re assuming a threat model where the AI has goals and motivations that are unpredictable and therefore risky, which is certainly the one that gets a lot of attention. But even if the AI’s goals and motivations can be perfectly controlled by its creators, you’re still at the mercy of the people who created the AI. In that respect it’s more of an arms race. And like many arms races, the goal might not necessarily be to outcompete everyone else so much as maintain a balance of power.
There’s no safe intelligence, so there’s no safe superintelligence. If you want safer superintelligence, you figure out how to augment the safest intelligence.
See MIRI https://intelligence.org/
"how someone so smart can be so naive"
Do you really think Ilya has not thought deeply about each and every one of your points here? There's plenty of answers to your criticisms if you look around instead of attacking.
7 replies →
> There isn’t going to be just one instance. Someone else will do the same
NK AI (!)
We're really not that far. I'd argue superintelligence has already been achieved, and it's perfectly and knowably safe.
Consider, GPT-4o or Claude are:
• Way faster thinkers, readers, writers and computer operators than humans are
• Way better educated
• Way better at drawing/painting
... and yet, appear to be perfectly safe because they lack agency. There's just no evidence at all that they're dangerous.
Why isn't this an example of safe superintelligence? Why do people insist on defining intelligence in only one rather vague dimension (being able to make cunning plans).
9 replies →
This all makes more sense when you realise it's Calvinism for programmers.
I think I heard that one before. Nuclear weapons are the Armageddon of nerds. Climate change is the Flood of the nerds. And so on.
Could you expand on this?
2 replies →
Calvinism for Transcendentalist techno-utopians -- an Asimovian Reformation of Singulatarianism
I think the idea is that a safe super intelligence would help solve those problems. I am skeptical because the vast majority are social coordination problems, and I don’t see how a machine intelligence no matter how smart can help with that.
So instead of a super intelligence either killing us all or saving us from ourselves, we’ll just have one that can be controlled to extract more wealth from us.
1 reply →
Social coordination problems exist within a specific set of constraints, and that set of constraints can itself be altered. For instance, climate change is often treated as a social coordination problem, but if you could produce enough energy cheaply enough, you could solve the greenhouse gas problem unilaterally.
4 replies →
I largely agree, although I do see how AI can help with social coordination problems, for example by helping elected leaders be more responsive to what their constituents need. (I spend a lot of my own time working with researchers at that intersection.) But social coordination benefits from energy research, too, and from biology research, and from the humanities, and from the arts. Computer science can't singlehandedly "solve" these problems any more than the other fields can; they are needed together, hence my gripe about total-orderings.
Exactly. Or who gets the results of its outputs. How do we prioritize limited compute?
4 replies →
are humans smarter than apes, and do humans do a better job at solving social coordination problems?
> I am skeptical because the vast majority are social coordination problems, and I don’t see how
Leadership.
By any means necessary I presume. If Russian propaganda helped get Trump elected, AI propaganda could help social coordination by influencing public perception of issues and microtargeting down to the individual level to get people on board.
1 reply →
It says «technical» problem, and probably implies that other technical problems could dramatically benefit from such achievement.
If you want a real technical revolution, you teach the masses how to code their own tailored software, and not just use abstractions and software built by people who sell software to the average user. What a shame we failed at that and are even sliding back in a lot of ways with plummeting technical literacy in smartphone-raised generations.
3 replies →
The blanket statements on the SSI homepage are pretty mediocre, and it is only the reputation of the founders that carries the announcement.
I think this quote at the end of this Bloomberg piece[0] gives more context,
> Sutskever says that the large language models that have dominated AI will play an important role within Safe Superintelligence but that it’s aiming for something far more powerful. With current systems, he says, “you talk to it, you have a conversation, and you’re done.” The system he wants to pursue would be more general-purpose and expansive in its abilities. “You’re talking about a giant super data center that’s autonomously developing technology. That’s crazy, right? It’s the safety of that that we want to contribute to.”
[0]: https://www.bloomberg.com/news/articles/2024-06-19/openai-co...
[0]: https://archive.is/ziMOD
So you're surprised when someone admits choosing to work on the problem they believe is the biggest and most important?
I guess they could be lying or badly disconnected from reality as you suggest. It would be far more interesting to read an argument for another problem being more valuable. It would be far cooler to hear about a plausible solution you're working on to solve that problem.
Yes, they see it as the top problem, by a large margin.
If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"
> the superlatives in these sorts of announcements really annoy me
I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.
Technical. He's saying it's the most important technical problem of our time.
Basically every problem is a "technical" problem in the year 2024 though? What problems out there don't have a solution that leverages technology?
2 replies →
[dead]
It's Palo Alto & Tel Aviv ordering that is total.
You don't need to survey every problem to feel some problem might be the most important one. If you think AGI/ASI is coming soon and extinction risks are high, you don't really need to order to see it's the most important problem.
It certainly is the most important technical problem of our time, if we end up developing such a system.
That conditional makes all the difference.
It's a hell of a conditional, though.
"How are all those monkeys flying out of my butt?" would be the important technical problem of our time, if and only if, monkeys were flying out of my butt.
It's still not a very important statement, if you downplay or omit the conditional.
Is "building safe superintelligence (SSI) is the most important technical problem of our time" full stop ?
Is it fuck.
2 replies →
[dead]
It is the most important problem of “our time” when you realize that the “our” here has the same meaning that it has in “our democracy”
C'mon. This one-pager is a recruiting document. One wants 'true believers' (intrinsically motivated) employees to execute the mission. Give Ilya some slack here.
Fair enough, and it's not worse than a lot of other product marketing messages about AI these days. But you can be intrinsically motivated by a problem without believing that other problems are somehow less important than yours.
exactly. and define safe. eg, is it safe (ie dereliction) to _not_ use ai to monitor dirty bomb threats? or more simple, CSAM?
In the context of super-intelligence, “safe” has been perfectly well defined for decades: “won't ultimately result in everyone dying or worse”.
You can call it hubris if you like, but don't pretend like it's not clear.
1 reply →
Love to see the traditional middlebrow dismissal as the top comment. Never change, HN.
> Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
It might be the case that the reason Ilya is “interested in” this problem (to the degree of dedicating almost his entire career to it) is exactly because he believes it’s the most important.
This. Next will hyperintelligence(R) /s
I believe that AGI is the last problem in computer science, so solving it solves all of the others. Then with AGI, we can solve the last remaining problems in physics (like unifying gravity with quantum mechanics), biology (administering gene therapy and curing death), etc.
But I do agree that innovations in tech are doing little or nothing to solve mass suffering. We had the tech to feed everyone in the world through farm automation by the 60s but chose not to. We had the tech in the 80s to do moonshots for AIDS, cancer, etc but chose not to. We had the tech in the 2000s to transition from fossil fuels to renewables but chose not to. Today we have the opportunity to promote world peace over continuous war but will choose not to.
It's to the point where I wonder how far innovations in tech and increases in economic productivity will get without helping people directly. My experience has been that the world chooses models like Dubai, Mexico City and San Francisco where skyscrapers tower over a surrounding homeless and indigent population. As long as we continue pursuing top-down leadership from governments and corporations, we'll see no change to the status quo, and even trends towards authoritarianism and fascism. It will take people at the bottom organizing to provide an alternate economic model before we have options like universal education/healthcare/opportunity and UBI from robot labor.
What gets me is that stuff like the ARC prize for AGI will "just work". As in, even if I had a modest stipend of a few thousand dollars per month to dabble in AI and come up with solutions the way I would for any other startup, certainly within 3 years, someone else would beat me to it. There simply isn't enough time now to beat the competition. Which is why I give AGI over 50% odds of arriving before 2030, where I used to think it was 2040 or 2050. The only thing that could stop it now is sabotage in the form of another global pandemic, economic depression or WWIII. Progress which threatens the power structures of the ultra wealthy is what drives the suffering that they allow to continue.
What a waste of an intelligence.
Pursuing artificial goal to solve a non existent problem to profit off meaningless hype around it.
World would have been better off if he made a decent alternative to k8s or invested his skills into curing cancer or at least protecting world from totalitarian governments and dangerous ideologies (if he wants to belong to vague generic cause).
You know, real problems, like the ones people used to solve back in the old days…
Nice absolute certainty you have there.
Has anyone got a good widely agreed definition of intelligence already?
Or at least hi quality and hi resolution understanding of what it is?
How can you really achieve (super|artificial|puper|duper)-intelligence then?
If not in your dreams and manipulated shareholders' expectations...
Until then yep, I'm quite certain we have a clear case of a naked king here.
There's always a bigger bubble. But now we're talking to infinity and beyond.
but would that stave off an impending recession?
By artificially postponing recession (you can't really avoid it) you postponing the next cycle of growth. While burning resources that could have helped you to survive it with less damage.
Any usage of the word "safe" without an accompanying precise definition of it is utter null and void.
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
https://www.safe.ai/work/statement-on-ai-risk, signed by Ilya Sutskever among others.
I clicked, hoping that "human extinction" was just the worst thing they were against. But that's the only thing. That leaves open a whole lot of bad stuff that they're OK with AI doing (as long as it doesn't kill literally everyone).
1 reply →
There are at least three competing definitions of the word:
There's the existential threat definition of "safe", put forth by Bostrom, Yudkowsky, and others. That's the idea that a superintelligent AI, or even one just incrementally smarter and faster than the humans working on AI, could enter a positive feedback loop in which it becomes overwhelmingly smarter and faster than humans, people can't control it, and it does unpredictable things.
There's the investor relations definition of "safe", which seems to be the one typically adopted by mission statements of OpenAI, Google, Meta, and others. That's (cynically) the fear that a chatbot with their branding on it promulgates culturally/ethically/morally unacceptable things it found in some dark corner the Internet, causing end users to do or think something reprehensible (and, not incidentally, causing really bad press in the process).
There's the societal harm definition of "safe", which is at first glance similar in to the investor relations safety definition, but which focuses on the specific judgements made by those filtering teams and the knock-on effects of access to these tools, like economic disruption to the job market.
Everyone seems to be talking past each other, dismissing or ignoring the concerns of other groups.
“””We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.”””
Cracked indeed
The phrase ‘crack team’ has military origins.
"Cracked team" has rather different connotations.
1 reply →
For fun I asked Claude (sonnet) to proofread that paragraph and it found no problem with "cracked team".
Interestingly I asked Opus and it did :-)
[dead]
[dead]
[dead]
[dead]
The NetBSD of AI? /s
[dead]
[flagged]
[flagged]
This is how you web
[flagged]
[flagged]
[flagged]
SSI technically. Super-Sized, you know?
[flagged]
[flagged]
[flagged]
the vision here is amazing, but I can't help but feel a bit uncertain about basing a "safe super intelligence" lab in an apartheid state that is currently genociding its indigenous population using variants of the very same technology
[flagged]
Right, true safe super intelligence could only be developed in countries that finished genociding their indigenous populations centuries ago.
Maybe it takes a nation (tragically) built on genocide and slavery to be able to recognize (and thus prevent) the same in an AI.
1 reply →
But basing it in US who launched the Afghanistan and Iraq wars is safe?
Well, yes¹: if it actually contributes to intelligent action, this is exactly the goal - you nailed it. Safer, more sensible action for individuals, communities, and statal entities. "Less mistakes in the future" - that would be a good motto.
(¹: whether basing a [successful in substance] "safe super intelligence" lab in an environment with imperfect record would improve safety.)
--
And now let us see if the sniper can come up with a good argument...
ungh, roots in israel, arent we boycotting them yet?
you might be. but if so, please delete so many of your daily apps and devices.
How can they speak of Safety when they are based partly in a colonialist settler entity that is committing a genocide and wanting to exterminate the indigenous population to make room for the Greater Zionist State.
I don't do business with Israeli companies while Israel is engaged in mass Extermination of a human population they treat as dogs.
[flagged]
Oh goodness, just what the world needs. Another self-righteous AI, something nobody actually wants.
Did you mean "Oh goodness, just what the world needs. Another self-righteous AI company"?
I think you must have, since we can't say anything about the "character" of their product yet.)