Comment by gonehome
3 years ago
If it’s not possible then yeah, there’s no risk. I just don’t find the arguments that it’s not possible very compelling.
We’re constrained in all sorts of ways because of biology, natural selection, energy, etc. I find it unlikely we just happen to be close to the max threshold.
If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
It is reasonable to assume that there is a maximum limit to local processing power.
The speed of light puts limits on how far information can move within a given latency threshold. As you expand a computational system's capacity you face unavoidable trade-offs between interconnection throughput, latency, and computational capacity.
We don't know how close to this maximum the human brain is. However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely.
> If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
Artificial human scale intelligence would already lead to massive shifts. However, the growth past that point could be incremental.
> "It is reasonable to assume that there is a maximum limit to local processing power."
As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso. It could be a very high limit; Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation
> "We don't know how close to this maximum the human brain is."
We don't, but we do know human eyes are not close to telescopes or microscopes, humans cannot sense radio waves directly at all, human speech is not close to the loudest noise, human memory can't compete with computer storage, human calculation ability can't compete with a scientific calculator, etc. Why would we assume that intelligence has any closer limits?
> "However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely."
Nature doesn't care if we have poor eyesight after age 40, we still make glasses - as far as nature is concerned there are diminishing returns, as far as we are concerned we like clear vision. We also like sunglasses, polarising lenses, swimming googles, safety goggles, magnifying glasses, loupes, night vision goggles, x-rays, thermal imaging, millimeter wave scanners, head-up displays, tele-vision; we haven't stopped trying to enhance our vision. Why rule out wanting to improve intelligence at least a lot further?
> As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso.
Yes, there are many criteria where engineering has trumped what evolution has produced. However there are many others where evolution has developed efficiency or finesse that we struggle to match. So far, intelligence falls in that later category.
> Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation
Those theoretical limits are interesting, but not really relevant as they are intended to find a value that can't be theoretically exceeded, not a practical uper bound.
So far, our understanding of intelligence requires significant communication between different regions of compute. As you try to scale this, you need to dedicate more and more volume to that communication and your average latency between compute regions goes up. Then comes the problem of heat dispersion, which also starts to consume more and more volume as the system scales.
These mean that if latency matters to intelligence (and our understanding of intelligence seems to indicate that it does), then there are real, practical limits on the scaling of intelligence.
> Why rule out wanting to improve intelligence at least a lot further?
I'm not ruling out the desire. I'm not even ruling out the possibility.
I am pointing out that designing intelligence seems to be a lot harder than launching satellites or building a telescope. Intelligence is hard and I've presented good reason to believe that it gets harder the more you try to scale it.
Thus it seems likely that iterative improvements in intelligence will become progressively harder in a way limits the potential for runaway growth.
This doesn't rule out the possibility of a paradigm shift in technology that significantly increases capacity but such a possibility also isn't guaranteed.
1 reply →
It’s not assuming we are near the limit of intelligence.
Let’s assume we can build something 3x as intelligent as a single person. What exactly can it do that a single person can’t? The thing is the world is filled with super human intelligence, groups of people can create things that are beyond any single person but they are still constrained by physical reality.
I don't exactly see how to define "3x as intelligent as a single person" so I'll conveniently define it as something that thinks 3x faster than a single person. A single such thing can talk to 3 persons simultaneously, and 100000 such things working together can talk to 300000 persons simultaneously.
Except you could also just use 300,000 human level intelligences to have those 300,000 conversations. So if having 300,000 conversations is the goal then super human intelligence doesn’t suddenly allow that to happen and may in fact make it harder if they take more than 3x the resources per intelligence.
A billion humans can't build a tree, but that's not because trees can't exist in physical reality. Something more intelligent than a human might a) be able to understand some part of this world which humans don't, and b) put it to some use that we aren't thinking about.
Might is doing some heavy lifting there, but of course humans have already gone past simply growing trees from seeds.
5 replies →