Comment by upwardbound

3 years ago

The most truly important sub-sector of the AI Safety field is what's officially known as AI Existential Safety, and which OpenAI employees jokingly call "AI NotKillEveryoneism".

AI Existential Safety is devoted to reducing two risks:

    "x-risk", short for Existential Risk, which is the risk that AI leads to our extinction.

    "s-risk", short for Suffering Risk, which is the risk that AI subjugates us and leads us to Matrix-like outcomes where we can neither escape nor die.

For an intro to how x-risk scenarios can come about, read about the "paperclip maximizer" thought experiment. Succinct explanation here: https://news.ycombinator.com/item?id=38344675

8 comments

upwardbound

nonethewiser 3 years ago

You have defined these sorts of safety concerns very clearly . Thank you.

This leads to the crux of the issue though - is that what the “more safety” camp is saying when they want to slow down AI development right now to ensure safety? Do they actually think we’re on the verge of that? Or is it lesser safety concerns like not answering questions about cross site scripting and not using training data with racial stereotypes baked in.

While the latter may be important it doesnt seem to constitute a reason to stop development. The x and s risks are reasons to stop development but I cant see a compelling reason to think we are remotely close to that.

upwardbound 3 years ago
> is the “more safety” camp saying to slow down AI development?
This is an EXCELLENT question and the answer is very very nuanced. Warning: the definitions of the terms I'm about to use are actively evolving in current discourse.
The following taxonomy reflects my own views, specifically with the inclusion of faction 4:
---
There are basically four competing factions.
(1) The "normal" faction, which includes Satya and almost all business people both in VC & on Wall Street. Normals say (through their actions and their investments, which both speak much louder than words) that we can deal with x-risk later, and right now let's make some money and continue life as normal. They focus their life's work on "buying a home", "saving for retirement", and maybe someday "giving back to their community", and other such comforting, familiar little platitudes of life as it was for our mom and dad. (2) The "decel" faction (short for "decelerate"), which includes most old-school AI safety folks such as Ilya, Helen, and Tasha. Sometimes you see these people with a "pause button" emoji or "stop sign" emoji in their Twitter name. (3) The "e/acc" faction (short for "effective accelerationists"). This faction is a mix of fanatical techno-utopians (like Yann LeCun and Andrew Ng), mixed with a bunch of Twitter people who post macho memes and have a "lol let's watch the world burn" sort of attitude. Those people are in my view very similar to the young people from 4chan who voted for Trump over Hillary in 2016 because they thought that a Trump presidency would be hilarious. (4) The newest faction doesn't even have a name. I've only heard it articulated by Greg Brockman so let's call it Brockism. This faction is very new and it actually has me reconsidering my own beliefs. Brockists believe that the safest way to reduce x-risk is to move as fast as possible with software development while moving as slowly as possible with semiconductor development. Basically, Brockman believes that semiconductors are already way too powerful and that we could accidentally stumble into artificial superintelligence by accidentally inventing a really good algorithm that's suddenly way smarter while still fitting in the limits of semiconductor technology as we know it (i.e., not requiring any fancy optical chips or quantum chips or memristor chips or 3D chips or any of the other ideas for what to do after Moore's Law soon stops progressing). The possibility that we could stumble into an accidental sudden leap in intelligence through a few lines of code is what Brockman believes is super dangerous and is what he calls the capabilities "overhang". As far as I know the Brockist ideology has only ever been articulated exactly once, which is in the final six minutes of this very interesting & heartwarming little TED talk:
https://youtu.be/C_78DM8fG6E?si=uIP2OIxV8dXAKr9B&t=1478
---
All in all:
- The "normal" and "e/acc" factions are both in my view stupidly naive, and both of them more or less advocate to follow standard Silicon Valley doctrine of "move fast, break things, get rich".
- The "decel" and "Brockist" factions both take x-risk super seriously, and agree on the need to restrict semiconductor development, but they have totally opposite views on whether AI software research should slow down or speed up.
For what happens next at OpenAI:
- In the political shake-up that just concluded, the "decel" faction lost everything, to the point where there is not even a single decel that I am aware of left standing in OpenAI leadership despite the fact that OpenAI was originally founded primarily by decels.
- Next, there will be an interesting and subtle three-way power struggle between the normals (Satya, + Sam?), Brockists (Brockman, + Sam?), and e/acc's (an ideology possibly held by some of the ML scientists).
- isaac_spzindel 3 years ago
  
  This is not entirely accurate. The majority of leaders from the old school AI folks and EA communities do not advocate attempting to stop AI development. Even Yudkowsky who is proabably the most doomer of them all, has advocated against it. This position does have a few supporters like Katja Grace, Kerry Vaughn, Adam Scholl etc but they are a minority.
  Most of the EAs want to stop and delay progress like Brockman said in the TED talk but know that it's not achievable in any significant way. Their views are much closer to Brockman and Altman. It's just that, even with the same broad view, there is disagreement in actions and strategies.
- upwardbound 3 years ago
  
  Significant empirical evidence for the Brockist position may be found in the accomplishments of the retro-computing "demoscene", which uses innovative software to produce computer graphics on par with the late 1990's on some of the very oldest personal computers.
  https://en.wikipedia.org/wiki/Demoscene
- nopinsight 3 years ago
  
  I've seen Sam's tweet that strongly suggests that he's in fact in or very close to camp 4), which you call Brockist.
  People often underestimate Sam as yet another capitalist. I believe his thinking is on quite another level and he does care for humanity. He doesn't seem to care about making much more money personally. He said on another occasion that he made money faster than he can spend it anyway.
  
  3 replies →