Comment by Retric
3 years ago
1. That assumes super intelligence is actually possible.
It’s clearly possible to have something more intelligent than humans, but that’s doesn’t mean you’re going to cross some threshold into a new category.
Take say weather prediction, more processing power doesn’t somehow make chaotic systems predictable from incomplete information.
And if superintelligence is possible then it assumes that it is useful. There are plenty of dumb systems that get good enough results. You can get better but it costs 10k times more for 1% gains. So why bother.
Also if superintelligence is useful then it also assumes that making intelligence ++ is easier than making intelligence. It might be that as you climb the intelligence ladder each next step takes super exponential intelligence to take such that we're already seeing the max.
Given that I haven't even seen a meaningful discussion on what intelligence even is, I tend to think superintelligence is probably not a threat.
I think superintelligence means progress. How fast is progress with a superintelligent being on earth? Could be exponential. We, humans, continue to make progress in everything that we do. Small breakthroughs compound on top of each other. For example, in order to make machines, we had to invent fire and iron. In order to make accurate weather forecasts, we had to invent the microchip, then supercomputers.
If a superintelligence can accelerate progress, there's no knowing what it can invent on top of each invention.
If it’s not possible then yeah, there’s no risk. I just don’t find the arguments that it’s not possible very compelling.
We’re constrained in all sorts of ways because of biology, natural selection, energy, etc. I find it unlikely we just happen to be close to the max threshold.
If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
It is reasonable to assume that there is a maximum limit to local processing power.
The speed of light puts limits on how far information can move within a given latency threshold. As you expand a computational system's capacity you face unavoidable trade-offs between interconnection throughput, latency, and computational capacity.
We don't know how close to this maximum the human brain is. However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely.
> If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
Artificial human scale intelligence would already lead to massive shifts. However, the growth past that point could be incremental.
> "It is reasonable to assume that there is a maximum limit to local processing power."
As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso. It could be a very high limit; Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation
> "We don't know how close to this maximum the human brain is."
We don't, but we do know human eyes are not close to telescopes or microscopes, humans cannot sense radio waves directly at all, human speech is not close to the loudest noise, human memory can't compete with computer storage, human calculation ability can't compete with a scientific calculator, etc. Why would we assume that intelligence has any closer limits?
> "However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely."
Nature doesn't care if we have poor eyesight after age 40, we still make glasses - as far as nature is concerned there are diminishing returns, as far as we are concerned we like clear vision. We also like sunglasses, polarising lenses, swimming googles, safety goggles, magnifying glasses, loupes, night vision goggles, x-rays, thermal imaging, millimeter wave scanners, head-up displays, tele-vision; we haven't stopped trying to enhance our vision. Why rule out wanting to improve intelligence at least a lot further?
2 replies →
It’s not assuming we are near the limit of intelligence.
Let’s assume we can build something 3x as intelligent as a single person. What exactly can it do that a single person can’t? The thing is the world is filled with super human intelligence, groups of people can create things that are beyond any single person but they are still constrained by physical reality.
I don't exactly see how to define "3x as intelligent as a single person" so I'll conveniently define it as something that thinks 3x faster than a single person. A single such thing can talk to 3 persons simultaneously, and 100000 such things working together can talk to 300000 persons simultaneously.
1 reply →
A billion humans can't build a tree, but that's not because trees can't exist in physical reality. Something more intelligent than a human might a) be able to understand some part of this world which humans don't, and b) put it to some use that we aren't thinking about.
7 replies →