Comment by mkaic

I'm excited to see a new thing in the Zero-Order Optimization (ZOO) world. I see most ZOO methods not as a "replacement for backprop" the way they are often marketed, but as a technique that can potentially work well in regimes where backprop is fundamentally weak. One of my favorite papers [0] in recent memory, for instance, is about using central-difference random gradient estimation (CD-RGE) to train large RNNs without using backprop-through-time. They also show it works decently for hard-to-optimize differentiable-neural-computers. Since reading that paper, I've had a lot of fun trying to apply this technique to settings I would describe as "a traditional differentiable neural network being applied in a non-traditional environment where you can't just call loss.backward and hope for the best". I am excited to try Dust out on my personal project now :)