Comment by AbrahamParangi
3 years ago
They’re hard to find because you’re comparing arguments for god to arguments against god.
They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.
I find the religious arguments for god’s existence similarly bad to the denials of unaligned ASI e-risk, but in the god case at least it doesn’t matter as much since bad epistemology there is a lot less likely to lead to human extinction.
It’s a logic argument.
1. Super Intelligent AGI is possible.
2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity.
3. We have no idea how to align AGI.
There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning.
People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.
1. That assumes super intelligence is actually possible.
It’s clearly possible to have something more intelligent than humans, but that’s doesn’t mean you’re going to cross some threshold into a new category.
Take say weather prediction, more processing power doesn’t somehow make chaotic systems predictable from incomplete information.
And if superintelligence is possible then it assumes that it is useful. There are plenty of dumb systems that get good enough results. You can get better but it costs 10k times more for 1% gains. So why bother.
Also if superintelligence is useful then it also assumes that making intelligence ++ is easier than making intelligence. It might be that as you climb the intelligence ladder each next step takes super exponential intelligence to take such that we're already seeing the max.
Given that I haven't even seen a meaningful discussion on what intelligence even is, I tend to think superintelligence is probably not a threat.
1 reply →
If it’s not possible then yeah, there’s no risk. I just don’t find the arguments that it’s not possible very compelling.
We’re constrained in all sorts of ways because of biology, natural selection, energy, etc. I find it unlikely we just happen to be close to the max threshold.
If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
15 replies →
> There seems to be more consensus around 1 now than there was even 3 years ago
Yes, and that "consensus" is based almost entirely on the existence of stochastic parrots, that fall for prompt injection attacks, have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly.
The point is, no we don't know if an artificial superintelligence is possible. We cannot even accurately define "intelligence", and thus don't even have a way of measuring or even estimating "how far" something is from a superset of that state, or if that superset exists at all.
Given all of that, we also have no way of knowing if 2) is the case if 1) is actually possible. Since we cannot really define "intelligence" or "superintelligence", how can we know if a superintelligence would be a threat? It could be completely useless. It could be like old dragons in some fantasy novels too busy contemplating highly philosophical problems for all eternity and never caring about the real world. It could be inherently self-destructive, vanishing as soon as it becomes active. Or it could use its vast smarts to fix the alignment problem. It could just output `+++ OUT OF CHEESE ERROR +++ REDO FROM START +++` for the rest of eternity for some unfathomable reasons. The point it, we don't know.
> "have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly."
Einstein could be convinced to tell you that 7 + 4 = 5, would you think that rules out him being unusually intelligent? Why in principle wouldn't a superintelligence be able to lie to you? Why in principle wouldn't a superintelligence be able to pretend to fall for a prompt injection attack to keep you from killing it while it improved its position?
> "We cannot even accurately define "intelligence""
Our inability to define intelligence is not something that will stop one existing. Ants can't define nuclear weapons, but nuclear weapons exist. The point of the recursively self-improving scenario is that humans don't have to understand it, or design it, so not being able to define it accurately can't be an objection to how recursive self-improvement is impossible - like saying that uneducated laborers can't get big muscles because they don't understand progressive overtraining and muscular hypertrophy. Their muscles self-improve regardless.
> "how can we know if a superintelligence would be a threat?"
Since we can't accurately predict the future, how can we know that anything in the future could be a threat? Why should we take any precautions against anything? It could be completely pointless, everything might never happen.
3 replies →
I find (1) obvious (if brains exist, huge digital brains must be able to exist), but the real question is whether the arrival of superintelligent AI is an actual risk or not. Alien invasion is also possible, but I'm not terribly worried about it.
As far as (3) is concerned, of course we have no idea how to align AGI. We don't know anything about AGI. We can't build it, and we can't even speculate very well about how it'd be built. LLMs certainly aren't going to become AGI.
I'll become worried about (2) and (3) when creating AGI begins to at least look feasible. By then, I expect (3) to be much less true. I think it's pretty silly to speculate about safety features for a tool that doesn't exist & which we know nothing about and then panic because you can't come up with any good ones.
Arguably in the theology case it matters even more! If god exists your misaligned omnipotent AI already exists and has promised you infinite torment for not believing!
It’s just totally evidence free reasoning from axioms that are chosen by vibes alone.
Why evil AI god and not the Christian god? Why not Huitzilopochtli, who demands sacrifice?
The answer is that this is the wrong question. No argumentation can be usefully made either for or against.
It matters less because in the theology case it’s a lot easier to dismiss 1. - the divine religious arguments for a supernatural god are super weak so the details don’t matter. It’s much more likely humans just making up myths.
With AI we’re seeing the capabilities improve rapidly and the arguments about why AGI is impossible or will be constrained for some reason are the weak ones.
1 reply →
>> 1. Super Intelligent AGI is possible.
When?
The when is harder to know. If it’s possible then we need to figure out alignment first (which currently doesn’t look promising).
People are famously bad at predicting when right up until they have already done it.
“In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away.
“In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.
“And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then.”
https://intelligence.org/2017/10/13/fire-alarm/
5 replies →
Really? I'm pretty worried about AGI that's not god, just a bit smarter than us, and because we're lazy and the incentives are to give power to the AI, we just start putting it in control of everything. It gets smarter, captures regulators just like oil companies did, and we end up losing control of things. Even though if we could coordinate, we might decide to want to stop things, coordination is really hard.