I think after you factor it in being on the extreme end of slow but wide (like laptop GPUs) and being a native representation of the neural network rather than a computed silicon approximation, and being more similar to a recurrent/continuous learning model rather than a huge all at once back prop, are there more tricks required?
My thought is you could probably set up something similar with today's technological capabilities but it would probably take decades to train and be expensive to develop which is commercially unacceptable.
Analog computation probably. Theoretically, technology like thermal wells and others that harness physics to find the local minimum energy state allow inference with multiple orders of magnitude improvements in size, energy, and speed. Rough back of the napkin gives SD-card size 1t parameter models on about 2.5 watts using thermal wells (cryogenic cooling required, afaik), achieving density approximately similar to NAND flash per parameter.
The simple explanation is that von Neumann architecture sucks for intelligence. It's not a von Neumann machine at all. It's an analog neural network (plus other systems in operation) with colocated memory.
A more far-out explanation is that some form of stochastic quantum computing is going on, which has been proposed at many levels by biologists. There's some work being done on this idea. It would not be the discrete qubit form we are trying to do, which is impossible in a noisy wet environment, but something analog and stochastic.
Both are possible.
Also note that our current chips, in spite of the tiny feature sizes and a ton of optimization work, are not even close to the lower limit for energy cost of computation. They still are not very efficient.
I think after you factor it in being on the extreme end of slow but wide (like laptop GPUs) and being a native representation of the neural network rather than a computed silicon approximation, and being more similar to a recurrent/continuous learning model rather than a huge all at once back prop, are there more tricks required?
My thought is you could probably set up something similar with today's technological capabilities but it would probably take decades to train and be expensive to develop which is commercially unacceptable.
Analog computation probably. Theoretically, technology like thermal wells and others that harness physics to find the local minimum energy state allow inference with multiple orders of magnitude improvements in size, energy, and speed. Rough back of the napkin gives SD-card size 1t parameter models on about 2.5 watts using thermal wells (cryogenic cooling required, afaik), achieving density approximately similar to NAND flash per parameter.
The simple explanation is that von Neumann architecture sucks for intelligence. It's not a von Neumann machine at all. It's an analog neural network (plus other systems in operation) with colocated memory.
A more far-out explanation is that some form of stochastic quantum computing is going on, which has been proposed at many levels by biologists. There's some work being done on this idea. It would not be the discrete qubit form we are trying to do, which is impossible in a noisy wet environment, but something analog and stochastic.
Both are possible.
Also note that our current chips, in spite of the tiny feature sizes and a ton of optimization work, are not even close to the lower limit for energy cost of computation. They still are not very efficient.