Comment by qoez
1 year ago
For things like machine learning I wonder how much extra performance could be squeezed out by simply working with continuous floating values on the analog level instead of encoding them as bits through a big indirect network of nands.
This is something that has been tried, basically constructing an analog matrix multiply/dot product and it gives reasonable power efficiency at into levels of precision. More precision and the analog accuracy leads to dramatic power efficiey losses (each bit is about 2x the power), so int8 is probably the sweet spot. The main issues are it is pretty inflexible and costly to design vs a digital int8 mac array, and hard to port to newer nodes, etc
I have wondered this and occasionally seen some related news.
Transistors can do more than on and off, there is also the linear region of operation where the gate voltage allows a proportional current to flow.
So you would be constructing an analog computer. Perhaps in operation it would resemble a meat computer (brain) a little more, as the activation potential of a neuron is some analog signal from another neuron. (I think? Because a weak activation might trigger half the outputs of a neuron, and a strong activation might trigger all outputs)
I don’t think we know how to construct such a computer, or how it would perform set computations. Like the weights in the neural net become something like capacitance at the gates of transistors. Computation is I suppose just inference, or thinking?
Maybe with the help of LLM tools we will be able to design such things. So far as I know there is nothing like an analog FPGA where you program the weights instead of whatever you do to an FPGA… making or breaking connections and telling LUTs their identity
You don't think we know how to construct an analog computer? We have decades of experience designing analog computers to run fire control systems for large guns.
https://maritime.org/doc/firecontrol/index.php
We have also a pretty decent amount of experience with (pulse/spiking) artificial neural networks in analog hardware, e.g. [1]. Very Energy efficient but yet hard to scale.
[1] https://www.kip.uni-heidelberg.de/Veroeffentlichungen/detail...
1 reply →
It's possible, but analog multiplication is hard and small analog circuits tend to be very noisy. I think there is a startup working on making an accelerator chip that is based on this principle, though.
There are optical accelerators on the market that - I believe - do that already, such as https://qant.com/photonic-computing/
You lose a lot of stability. Each operation's result is slightly off, and the error accumulates and compounds. For deep learning in particular, many operations are carried in sequence and the error rates can become inacceptable.
Deep learning is actually very tolerant to imprecision, which is why it is typically given as an application for analog computing.
It is already common practice to deliberately inject noise into the network (dropout) at rates up to 50% in order to prevent overfitting.
Isn't it just for inference? Also, differentiating thru an analog circuit looks... interesting. Keep inputs constant, wiggle one weight a bit, store how the output changed, go to the next weight, repeat. Is there something more efficient, I wonder.
1 reply →
TLC,QLC,MLC in ssd is it. so it is used already. and it gives you limits of current technology.
>*TLC,QLC,MLC"
For those unaware of these acronyms (me):
TLC = Triple-Layer Cell
QLC = Quad-Level Cell
MLC = Multi-Level Cell
For those unaware, MLC used to mean mean Two-level cell.
Quad-level is the current practical maximum.