Comment by gonehome
3 years ago
Yeah, seeing how bad the rebuttals are to superintelligence e-risk really does make me feel like we’re doomed.
They’re frustratingly dumb, I’d like to see some good arguments or steelman but they’re honestly very hard to find.
Ultimately it seems like there’s nothing most individuals can do so just live your life as you would and hope the timeline is farther out than it seems.
Worse than the Cold War nuclear risk imo, at least in that case it was possible for humans to stupidly build thousands and then decide not to use them (and it’s relatively easy to restrict uranium access/control the development of nukes). Not really the case with superintelligent AGI.
People have a bad heuristic for what tailrisk exists. They think extinction talk is impossible or crazy. We won’t get the opportunity to mess up and try again in this case.
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
They’re hard to find because you’re comparing arguments for god to arguments against god.
They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.
I find the religious arguments for god’s existence similarly bad to the denials of unaligned ASI e-risk, but in the god case at least it doesn’t matter as much since bad epistemology there is a lot less likely to lead to human extinction.
It’s a logic argument.
1. Super Intelligent AGI is possible.
2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity.
3. We have no idea how to align AGI.
There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning.
People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.
1. That assumes super intelligence is actually possible.
It’s clearly possible to have something more intelligent than humans, but that’s doesn’t mean you’re going to cross some threshold into a new category.
Take say weather prediction, more processing power doesn’t somehow make chaotic systems predictable from incomplete information.
18 replies →
> There seems to be more consensus around 1 now than there was even 3 years ago
Yes, and that "consensus" is based almost entirely on the existence of stochastic parrots, that fall for prompt injection attacks, have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly.
The point is, no we don't know if an artificial superintelligence is possible. We cannot even accurately define "intelligence", and thus don't even have a way of measuring or even estimating "how far" something is from a superset of that state, or if that superset exists at all.
Given all of that, we also have no way of knowing if 2) is the case if 1) is actually possible. Since we cannot really define "intelligence" or "superintelligence", how can we know if a superintelligence would be a threat? It could be completely useless. It could be like old dragons in some fantasy novels too busy contemplating highly philosophical problems for all eternity and never caring about the real world. It could be inherently self-destructive, vanishing as soon as it becomes active. Or it could use its vast smarts to fix the alignment problem. It could just output `+++ OUT OF CHEESE ERROR +++ REDO FROM START +++` for the rest of eternity for some unfathomable reasons. The point it, we don't know.
4 replies →
I find (1) obvious (if brains exist, huge digital brains must be able to exist), but the real question is whether the arrival of superintelligent AI is an actual risk or not. Alien invasion is also possible, but I'm not terribly worried about it.
As far as (3) is concerned, of course we have no idea how to align AGI. We don't know anything about AGI. We can't build it, and we can't even speculate very well about how it'd be built. LLMs certainly aren't going to become AGI.
I'll become worried about (2) and (3) when creating AGI begins to at least look feasible. By then, I expect (3) to be much less true. I think it's pretty silly to speculate about safety features for a tool that doesn't exist & which we know nothing about and then panic because you can't come up with any good ones.
Arguably in the theology case it matters even more! If god exists your misaligned omnipotent AI already exists and has promised you infinite torment for not believing!
It’s just totally evidence free reasoning from axioms that are chosen by vibes alone.
Why evil AI god and not the Christian god? Why not Huitzilopochtli, who demands sacrifice?
The answer is that this is the wrong question. No argumentation can be usefully made either for or against.
2 replies →
>> 1. Super Intelligent AGI is possible.
When?
6 replies →
Really? I'm pretty worried about AGI that's not god, just a bit smarter than us, and because we're lazy and the incentives are to give power to the AI, we just start putting it in control of everything. It gets smarter, captures regulators just like oil companies did, and we end up losing control of things. Even though if we could coordinate, we might decide to want to stop things, coordination is really hard.
Personally I find people with the superintelligence fetish frustratingly dumb too.
Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.
And that's before we even think about whether it would even have the motivation.
I think there's a reason humans plateaued where they did, very intelligent people often really struggle in the real world.
> Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.
When the "people with the superintelligence fetish" worry about AI, at least they provide a reason why it's likely that AI will make humans extinct.
What is your heuristic to determine what is likely?
The plateau of human intelligence isn't especially relevant given we are the product of evolution rather than engineering.
Personally,I think we stand a pretty decent chance of making a horrible mess of this all well before reaching AGI. However, a machine that could improve itself would not be limited in anyway that humans are; and I think we know little of how it would behave.
That is assuming it can't be controlled; otherwise it will behave as directed by whoever controls it.