Comment by jodrellblank
3 years ago
> "It is reasonable to assume that there is a maximum limit to local processing power."
As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso. It could be a very high limit; Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation
> "We don't know how close to this maximum the human brain is."
We don't, but we do know human eyes are not close to telescopes or microscopes, humans cannot sense radio waves directly at all, human speech is not close to the loudest noise, human memory can't compete with computer storage, human calculation ability can't compete with a scientific calculator, etc. Why would we assume that intelligence has any closer limits?
> "However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely."
Nature doesn't care if we have poor eyesight after age 40, we still make glasses - as far as nature is concerned there are diminishing returns, as far as we are concerned we like clear vision. We also like sunglasses, polarising lenses, swimming googles, safety goggles, magnifying glasses, loupes, night vision goggles, x-rays, thermal imaging, millimeter wave scanners, head-up displays, tele-vision; we haven't stopped trying to enhance our vision. Why rule out wanting to improve intelligence at least a lot further?
> As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso.
Yes, there are many criteria where engineering has trumped what evolution has produced. However there are many others where evolution has developed efficiency or finesse that we struggle to match. So far, intelligence falls in that later category.
> Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation
Those theoretical limits are interesting, but not really relevant as they are intended to find a value that can't be theoretically exceeded, not a practical uper bound.
So far, our understanding of intelligence requires significant communication between different regions of compute. As you try to scale this, you need to dedicate more and more volume to that communication and your average latency between compute regions goes up. Then comes the problem of heat dispersion, which also starts to consume more and more volume as the system scales.
These mean that if latency matters to intelligence (and our understanding of intelligence seems to indicate that it does), then there are real, practical limits on the scaling of intelligence.
> Why rule out wanting to improve intelligence at least a lot further?
I'm not ruling out the desire. I'm not even ruling out the possibility.
I am pointing out that designing intelligence seems to be a lot harder than launching satellites or building a telescope. Intelligence is hard and I've presented good reason to believe that it gets harder the more you try to scale it.
Thus it seems likely that iterative improvements in intelligence will become progressively harder in a way limits the potential for runaway growth.
This doesn't rule out the possibility of a paradigm shift in technology that significantly increases capacity but such a possibility also isn't guaranteed.
What does efficiency or finesse have to do with it? Motorocycles still exist despite needing a global supply chain and looking blocky and chunky. An intelligence that needs a datacenter and a multi-megawatt power supply could still exist.
> "average latency between compute regions goes up."
This Google'd article[1] says a brain could have 20mS of latency from front to back. We can ping over a hundred miles in that time with today's packet switched public networks, and light can travel 3,750 miles in that time. That's enough space to make a big 'brain' computer.
> "Then comes the problem of heat dispersion"
Biological brains have to stay alive, and have to be energy efficient because food can be scarce. Computers can be cooled with liquid nitrogen[2] without dying, or be under the ocean, in the Arctic, in space, even assuming a superintelligence couldn't come up with new architectures or new cooling methods. Infinite growth, or growth up to the upper bounds of theoretical physics is unlikely, I grant you that, but (assuming it can be done with computers at all) there seems to be a lot of room for a lot of growth.
[1] https://discoverysedge.mayo.edu/2023/03/09/understanding-the...
[2] https://www.pcgamer.com/overclocking-a-cpu-to-7-ghz-with-the...