Comment by gonehome
3 years ago
I find the religious arguments for god’s existence similarly bad to the denials of unaligned ASI e-risk, but in the god case at least it doesn’t matter as much since bad epistemology there is a lot less likely to lead to human extinction.
It’s a logic argument.
1. Super Intelligent AGI is possible.
2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity.
3. We have no idea how to align AGI.
There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning.
People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.
1. That assumes super intelligence is actually possible.
It’s clearly possible to have something more intelligent than humans, but that’s doesn’t mean you’re going to cross some threshold into a new category.
Take say weather prediction, more processing power doesn’t somehow make chaotic systems predictable from incomplete information.
And if superintelligence is possible then it assumes that it is useful. There are plenty of dumb systems that get good enough results. You can get better but it costs 10k times more for 1% gains. So why bother.
Also if superintelligence is useful then it also assumes that making intelligence ++ is easier than making intelligence. It might be that as you climb the intelligence ladder each next step takes super exponential intelligence to take such that we're already seeing the max.
Given that I haven't even seen a meaningful discussion on what intelligence even is, I tend to think superintelligence is probably not a threat.
I think superintelligence means progress. How fast is progress with a superintelligent being on earth? Could be exponential. We, humans, continue to make progress in everything that we do. Small breakthroughs compound on top of each other. For example, in order to make machines, we had to invent fire and iron. In order to make accurate weather forecasts, we had to invent the microchip, then supercomputers.
If a superintelligence can accelerate progress, there's no knowing what it can invent on top of each invention.
If it’s not possible then yeah, there’s no risk. I just don’t find the arguments that it’s not possible very compelling.
We’re constrained in all sorts of ways because of biology, natural selection, energy, etc. I find it unlikely we just happen to be close to the max threshold.
If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
It is reasonable to assume that there is a maximum limit to local processing power.
The speed of light puts limits on how far information can move within a given latency threshold. As you expand a computational system's capacity you face unavoidable trade-offs between interconnection throughput, latency, and computational capacity.
We don't know how close to this maximum the human brain is. However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely.
> If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
Artificial human scale intelligence would already lead to massive shifts. However, the growth past that point could be incremental.
3 replies →
It’s not assuming we are near the limit of intelligence.
Let’s assume we can build something 3x as intelligent as a single person. What exactly can it do that a single person can’t? The thing is the world is filled with super human intelligence, groups of people can create things that are beyond any single person but they are still constrained by physical reality.
10 replies →
> There seems to be more consensus around 1 now than there was even 3 years ago
Yes, and that "consensus" is based almost entirely on the existence of stochastic parrots, that fall for prompt injection attacks, have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly.
The point is, no we don't know if an artificial superintelligence is possible. We cannot even accurately define "intelligence", and thus don't even have a way of measuring or even estimating "how far" something is from a superset of that state, or if that superset exists at all.
Given all of that, we also have no way of knowing if 2) is the case if 1) is actually possible. Since we cannot really define "intelligence" or "superintelligence", how can we know if a superintelligence would be a threat? It could be completely useless. It could be like old dragons in some fantasy novels too busy contemplating highly philosophical problems for all eternity and never caring about the real world. It could be inherently self-destructive, vanishing as soon as it becomes active. Or it could use its vast smarts to fix the alignment problem. It could just output `+++ OUT OF CHEESE ERROR +++ REDO FROM START +++` for the rest of eternity for some unfathomable reasons. The point it, we don't know.
> "have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly."
Einstein could be convinced to tell you that 7 + 4 = 5, would you think that rules out him being unusually intelligent? Why in principle wouldn't a superintelligence be able to lie to you? Why in principle wouldn't a superintelligence be able to pretend to fall for a prompt injection attack to keep you from killing it while it improved its position?
> "We cannot even accurately define "intelligence""
Our inability to define intelligence is not something that will stop one existing. Ants can't define nuclear weapons, but nuclear weapons exist. The point of the recursively self-improving scenario is that humans don't have to understand it, or design it, so not being able to define it accurately can't be an objection to how recursive self-improvement is impossible - like saying that uneducated laborers can't get big muscles because they don't understand progressive overtraining and muscular hypertrophy. Their muscles self-improve regardless.
> "how can we know if a superintelligence would be a threat?"
Since we can't accurately predict the future, how can we know that anything in the future could be a threat? Why should we take any precautions against anything? It could be completely pointless, everything might never happen.
> Einstein could be convinced to tell you that 7 + 4 = 5
I think Einstein would have laughed at me if I tried to convince him to do that.
Because other than an LLM, Einstein knew what these symbols denote, what their relation to reality is, and how math works. Einstein didn't mimick math by completing sequences of tokens, and relying on humans antropomorphizing the sequence completion engines output to an actual understanding of the topic.
> Ants can't define nuclear weapons, but nuclear weapons exist.
Ants also cannot build nuclear weapons, nor create anything that would make the emergence of nukes any more likely, among other things because they don't have the ability to define them. So if we accept this premise, then the discussion is moot: We can be fairly certain that we are the most technologically capable entities on this world, so unless we can understand a technological creation to the extend that we can bring it about, nothing else will.
In short: If we are ants to the superintelligence, then we have nothing to worry about, because we likely lack the understanding and ability to create it, or even something that could act as its precursor. If we are not ants, then we should be able to predict when this can happen.
> Since we can't accurately predict the future
We can accurately predict a lot of things. Global warming is an example. And the things that we can predict, and determine how likely they are, we can and should prepare for.
AI doomsday preparation demands the exact opposite: That we prepare for something that we cannot predict, and cannot demonstrate if it is possible, or how likely it is. That's like asking to prepare for an ice age. Theoretically an ice age is possible on this planet, however nothing we can see, measure and demonstrate right now, supports the prediction that an ice age is about to destroy us.
2 replies →
I find (1) obvious (if brains exist, huge digital brains must be able to exist), but the real question is whether the arrival of superintelligent AI is an actual risk or not. Alien invasion is also possible, but I'm not terribly worried about it.
As far as (3) is concerned, of course we have no idea how to align AGI. We don't know anything about AGI. We can't build it, and we can't even speculate very well about how it'd be built. LLMs certainly aren't going to become AGI.
I'll become worried about (2) and (3) when creating AGI begins to at least look feasible. By then, I expect (3) to be much less true. I think it's pretty silly to speculate about safety features for a tool that doesn't exist & which we know nothing about and then panic because you can't come up with any good ones.
Arguably in the theology case it matters even more! If god exists your misaligned omnipotent AI already exists and has promised you infinite torment for not believing!
It’s just totally evidence free reasoning from axioms that are chosen by vibes alone.
Why evil AI god and not the Christian god? Why not Huitzilopochtli, who demands sacrifice?
The answer is that this is the wrong question. No argumentation can be usefully made either for or against.
It matters less because in the theology case it’s a lot easier to dismiss 1. - the divine religious arguments for a supernatural god are super weak so the details don’t matter. It’s much more likely humans just making up myths.
With AI we’re seeing the capabilities improve rapidly and the arguments about why AGI is impossible or will be constrained for some reason are the weak ones.
> we’re seeing the capabilities improve rapidly
Yes, but towards what? How do we know that, say, Transformer based LLMs are closer to superintelligence than earlier architectures?
To make such an assumption, there would need to be something that we could measure to track the process. To the best of my knowledge, there is no accurate definition of intelligence, nor superintelligence.
So how would we know where on the scale of [intelligent-------superintelligent] a given system is, or whether it even is on that scale?
>> 1. Super Intelligent AGI is possible.
When?
The when is harder to know. If it’s possible then we need to figure out alignment first (which currently doesn’t look promising).
People are famously bad at predicting when right up until they have already done it.
“In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away.
“In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.
“And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then.”
https://intelligence.org/2017/10/13/fire-alarm/
>> The when is harder to know.
Next question:
How?
4 replies →