Comment by htk
3 days ago
So Microsoft is actually using 2 bits instead of 1.58. In this case they could represent -1, 0, 1, 2. As inhibitory synapses account for 20%-30%, this could map well to how biological brains are structured.
Does that make sense?
Can you explain your third statement?
> As inhibitory synapses account for 20%-30%, this could map well to how biological brains are structured.
In the human brain most synapses are indeed excitatory, while a minority is inhibitory.
No concise HN comment will give you a complete picture of whats currently known about the human brain, so a platitude necessarily follows:
We call the nearly touching interfaces between neurons synapses, small packets / droplets of neurotransmitter are sent across this interface from the source to the target neuron. Such signals can be excitatory (promote the probability of excitation of the target firing soon) or inhibitory (inhibits the probability of the target firing soon). There are 2 types of sensitive areas on your average neuron: the dendrites (long branching tentacles, that receive excitatory signals) and the cell body where all the signals are accumulated to a local instantaneous "sum" is also sensitive to synaptic activation, but the synapses on the cell body are inhibitory, when sufficiently inhibited the neuron will refuse to fire its axons, so the inhibitory synapses on the cell body can gate the cumulative signal and prevent it from triggering this neuron temporarily. If the neuron does fire, this propagates along the axons (another type of branching tentacles, which lead to yet other neurons, sometimes touching them excitatorily at their dendrite, sometimes touching a neuron inhibitorily at their cell body.
I hope that helped?
It is really truly incredible that this mess of microscopic meat plumbing encodes everything we see, think, and do. Terrifying and amazing all at once.
I did not realize all the dendritic synapses were excitatory, I always thought it depended on the specific neurotransmitters released. Thanks, this is cool. I am curious what will happen when we build LLMs that have the equivalent of chemical diffusions between synaptic release areas as well as the temporality of spiking neural nets.