Comment by famouswaffles

>A biological neural network is certainly not differentiab

Biology is biology and has its constraints. Doesn't necessarily mean a biologically plausible optimizer would be the most efficient or correct way in silicon.

>If the thing we want to build is not realizable with this technique, why can't we move on from it?

All the biologically plausible optimizers we've fiddled with (and we've fiddled with quite a lot) just work (results wise) like gradient descent but worse. We've not "moved on" because gradient descent is and continues to be better.

>Evolutionary techniques can explore impossibly large, non-linear problem spaces.

Sure, with billions of years (and millions of concurrent experiments) on the table.