Every few years, a derivative-free neural network optimization algorithm gets some hype. I'd bet my life savings that none of them ever make an impact.
Derivative-free optimization can be useful for genuinely discontinuous objectives [1], but common neural network objectives are smooth and/or Lipschitz.
The gradient is useful. Instead of trying random directions and hoping that one of them is an improvement, it tells you where to go. The more parameters you have, the more useful it becomes.
A strict complexity gap between gradient-based and derivative-free Lipschitz convex optimization has been suspected for decades and recently proved (using AI, [2]). Neural net optimization is nonconvex, but not radically different.
IMO, a more promising direction is gradient-based optimizers specialized to the neural network structure, like Muon [3].
It's still important to support research in this direction cause us meatbags cost a LOT less energy to train than GPTs even if you assume that it takes 30 years to train a PhD. Our current training methods honestly leave a lot to be desired. We almost certainly dont do full backpropagation.
You need to factor in the cost of training all those PhDs that never end up producing much of interest though you don't get to pick the best afterwards and claim all it took was to train him/her. Plus, once trained, it's productive only for 6 subjective hours per day (including weekends and holidays). Might have costed gazillions to train a frontier LLM but it works for millions of hours/day.
Even for non-smooth or discontinuous objectives, I'd still reach for methods that use gradient-like information over zeroth-order methods. For non-smooth objectives, Clarke-generalized subdifferentials have been pretty effective outside of ML, and have been used in automatic differentiation contexts at least 10-15 years ago. A carelessly quick literature search suggested conservative gradients, too.
For discontinuous objectives, I know there's been work on using envelope approximations, but the little I'm aware of in that work was in low-dimensional settings where the structure of the discontinuity was known explicitly. On the other extreme, lack of continuity comes up all the time in infinite-dimensional, PDE-constrained optimization, and some methods rely on tangent cones or various generalized notions of subdifferentiability (e.g., Mordukhovich, Bouligand) to demonstrate convergence. Admittedly, that work was somewhat outside my area of expertise, so I may be getting the details there slightly wrong, but the broad point stands that even in those settings, some directional information can be obtained and used profitably without resorting to zeroth-order methods.
Kind of like random projections. Pre ChatGPT there would every now and then come a paper that states something along the lines of: instantiate a large randomly populated matrix, multiply by the input, and win! It kinda makes sense because you "stretch out" the space and separation boundaries become easier, but I have never seen it fully utilised in production in any meaningful way. Anyone remember the DANs (Deep Averaging Networks)?
It is like Monte Carlo integration and the curse of dimensionality. Yes, you can use this for basically anything but it pays off only if the object you are trying to integrate is multidimensional, otherwise traditional methods outperforms
Isn't the 'we need gradients' a foregone conclusion?
Gradients can be calculated numerically, meaning that any method that samples the cost function and makes optimization decisions based on that can actually compute gradients if it needs to.
If it's cheaper then I guess it will might get practical. Also more likely that mind uses simpler techniques more similar to these.
I wonder though if someone tested mutating training objective though, like keeping original loss/goal and somehow defining loss differently and then comparing against original. This intuitively feels like how mind tries to handle difficult tasks.
"A strict complexity gap between gradient-based and derivative-free Lipschitz convex optimization" has been known for decades. It's covered in standard textbooks like Nesterov's "Introductory lectures on convex optimization". That AI result is about establishing the gap in a fairly niche accuracy regime.
Thanks for the correction, I had a feeling that should be true but the headline was at top of search results and I'm not close enough to the field to know off the top of my head.
> Derivative-free optimization can be useful for genuinely discontinuous objectives [1], but common neural network objectives are smooth and/or Lipschitz.
Like for derivatives, there is a chain rule for Boolean variations, so you can use something like backpropagation, but without needing any expensive floating point math. Though I don't think this has been used much so far. There must be some other downside.
I'm skeptical as to whether zeroth-order methods really lend themselves to a Bitter Lesson argument. First-order methods don't explore the loss landscape optimally, but the loss function tends to be nonconvex, and zeroth-order methods don't address that issue head on. Dust smooths, and so do applicable first-order methods. Remove the nonconvexity issue, and I suspect Dust's purported advantages evaporate (based on published theoretical work), so it's pretty odd to me that the paper never discusses convexity.
I could buy that this method scales better than previous zeroth-order methods, and that's interesting, but it doesn't seem like enough of a moat to keep improved first-order methods from drinking its milkshake, except in cases where a zeroth-order method is already a primary option: the network needs to call a simulator that doesn't expose gradient-like information. (In cases where gradients don't exist, I'd still argue for other options, e.g., Clarke-generalized gradients where applicable, so long as those can be computed with the available information. I know this technology has been published for automatic differentiation, so I would imagine it could be incorporated into backprop and used with a suitable optimization algorithm.)
Orders-of-magnitude improvements in compute efficiency are needed to become a practical replacement for backprop… but those improvements are coming.
As a mixture, could activation-space search produce useful teaching targets for backprop?
Zeroth-order search would discover candidates, first-order learning would consolidate them. The potentially valuable step is converting a sparse judgment into a reusable training target.
I'm not sure I follow why your argument changes the relevance of convexity? If the problems were convex, I don't think we'd be having this discussion -- first-order methods would tend to win.
I'm excited to see a new thing in the Zero-Order Optimization (ZOO) world. I see most ZOO methods not as a "replacement for backprop" the way they are often marketed, but as a technique that can potentially work well in regimes where backprop is fundamentally weak. One of my favorite papers [0] in recent memory, for instance, is about using central-difference random gradient estimation (CD-RGE) to train large RNNs without using backprop-through-time. They also show it works decently for hard-to-optimize differentiable-neural-computers. Since reading that paper, I've had a lot of fun trying to apply this technique to settings I would describe as "a traditional differentiable neural network being applied in a non-traditional environment where you can't just call loss.backward and hope for the best". I am excited to try Dust out on my personal project now :)
> There are many interesting open questions. The first is whether, and how, Dust can find better directions than backprop’s first-order gradient
Both algorithms are bound by the same Pareto frontier based on the Empirical Risk Minimisation Principle, so they’re already on the same trajectory.
Interestingly backprop is limited by conditioning of the Hessian matrix in order to converge (differentiate correctly). So removing this limitation is actually a great step.
I’m excited to see a comeback of evolutionary methods because they’re much more general, albeit costly and naive.
We’re now very close to what can be described best as brute forcing the Pareto frontier out of our datasets. Not sure that’s what we want but I have no better ideas either.
Even though this is way more expensive than backprop, could a hybrid approach where you fine tune an existing checkpoint that's been backpropped unlock further gains?
It would be cool to apply this to different stages and see if that affects the learning trajectory
Intereting. There was a completely different take on how to "do" back propagation using optics at YC papers video the other day(I can't find the video). The technique presented diffusion networks with optics. If a hardware technique to make computation more efficient emerges, a lot of methods like this could apply no?
I would like to see wall clock time, energy, peak memory and downstream quality compared at equal loss. Until the effiency gap closes, this is an interesting research direction.
For me credit assignment without a global clock is extremely important, i believe it's a necessary step towards end to end training of models In-materio.
Not necessarily, backprop is highly parallelizable since it is just a bunch of matrix mults.
Something like Dust skips the backward pass on backprop. But other techniques like
Neural Predictive Coding can be completely asynchronous, each "weight" can fire independent of those far away from it. Innocenti, et. al have shown that NPC gradients converge to backprop within a certain "regime".
The win with asynchronous techniques like NPC is that you do not need the extreme co-ordination that backprop requires and hence should be computationally much easier given the right device.
Although at this point the industry has so much money in the forward-backward pass system that I doubt a backprop successor would win unless someone makes NPC hardware feasible and can prove scaling up to billions of params
The reason why I don't see the promise for ML-only applications is that the coordination backprop requires comes very cheap to us.
"Much easier given the right device" - the "right" there just isn't shaped like the devices we actually build. And the price of "not having backprop" is usually expending more FLOPs, getting worse sample efficiency, etc.
The biggest "device" that doesn't do backprop is the brain, and that's because the brain doesn't have the connectivity or the coordination to pull it off. Both of those are "expensive" for something like it to implement. Cheap for us though. We aren't stuck with neurons that only get locally available information and have to implement learning rules based on that. So, skill issue?
Knowing nothing about this, I wonder if it could be useful in situations where we can’t reliably sync with all the workers. Something like folding@home, where all the workers are just shaking weights and if one of them finds a winner it uploads to the central server?
I think Jeff Dean is right in that we will see much more specialised silicon in the future.
If something more bio inspired ie. predictive coding and in-memory compute fundamentally makes continual learning and much lower energy consumption possible there will be specialised hardware for it at some point
FWIW I think the brain has multiple “learning rules” and operates at multiple timescales
Would these alternatives to backprop make it more feasible to have constant live-training going on in a model? Giving it something akin to neuro-plasticity?
At massive scales 0th order methods will parallelize better than backprop especially along depth, u can train very deep models pipeline parallel without bubbles
This is super yawn-worthy. Instead of backprop for the exact gradient you can run forward passes a thousand times with perturbed weights and get a Monte Carlo estimate of the gradient. Not very clever. Extremely NOT useful.
But I guess the industry is littered with techniques for computing the same thing but vastly slower that some people find interesting. Homomorphic encryption. Zero knowledge proofs. Blockchain computing. Except in those cases there might be a legitimate reason to use it occasionally.
Moving beyond traditional backpropagation for transformer pretraining opens up fascinating avenues for alternative learning dynamics and architectural efficiency.
- isnt the whole point of back propoagation so that you dont guess weights by brute forcing them since that is computationally infeasible once you go beyond a dozen weights?
- if you dont use backpropogation, how exactly are the initial weights assigned them if they are not random values?
Every few years, a derivative-free neural network optimization algorithm gets some hype. I'd bet my life savings that none of them ever make an impact.
Derivative-free optimization can be useful for genuinely discontinuous objectives [1], but common neural network objectives are smooth and/or Lipschitz.
The gradient is useful. Instead of trying random directions and hoping that one of them is an improvement, it tells you where to go. The more parameters you have, the more useful it becomes.
A strict complexity gap between gradient-based and derivative-free Lipschitz convex optimization has been suspected for decades and recently proved (using AI, [2]). Neural net optimization is nonconvex, but not radically different.
IMO, a more promising direction is gradient-based optimizers specialized to the neural network structure, like Muon [3].
[1] https://arxiv.org/abs/2202.00817
[2] https://arxiv.org/abs/2607.13335
[3] https://jeremybernste.in/writing/deriving-muon
It's still important to support research in this direction cause us meatbags cost a LOT less energy to train than GPTs even if you assume that it takes 30 years to train a PhD. Our current training methods honestly leave a lot to be desired. We almost certainly dont do full backpropagation.
I’m not sure that’s true. We have literal billions of years of training.
3 replies →
I'm sure we don't literally do backpropagation, but differential equations that settle toward stationary states show up a lot in biology
You need to factor in the cost of training all those PhDs that never end up producing much of interest though you don't get to pick the best afterwards and claim all it took was to train him/her. Plus, once trained, it's productive only for 6 subjective hours per day (including weekends and holidays). Might have costed gazillions to train a frontier LLM but it works for millions of hours/day.
PHD inference doesn't scale, though.
I think it's fun back-of-the-napkin sometimes to compare meat to matmuls but ultimately fallacious to its core, making it an intellectual tarpit.
It's much harder to argue with the math and empirical results.
1 reply →
Even for non-smooth or discontinuous objectives, I'd still reach for methods that use gradient-like information over zeroth-order methods. For non-smooth objectives, Clarke-generalized subdifferentials have been pretty effective outside of ML, and have been used in automatic differentiation contexts at least 10-15 years ago. A carelessly quick literature search suggested conservative gradients, too.
For discontinuous objectives, I know there's been work on using envelope approximations, but the little I'm aware of in that work was in low-dimensional settings where the structure of the discontinuity was known explicitly. On the other extreme, lack of continuity comes up all the time in infinite-dimensional, PDE-constrained optimization, and some methods rely on tangent cones or various generalized notions of subdifferentiability (e.g., Mordukhovich, Bouligand) to demonstrate convergence. Admittedly, that work was somewhat outside my area of expertise, so I may be getting the details there slightly wrong, but the broad point stands that even in those settings, some directional information can be obtained and used profitably without resorting to zeroth-order methods.
Kind of like random projections. Pre ChatGPT there would every now and then come a paper that states something along the lines of: instantiate a large randomly populated matrix, multiply by the input, and win! It kinda makes sense because you "stretch out" the space and separation boundaries become easier, but I have never seen it fully utilised in production in any meaningful way. Anyone remember the DANs (Deep Averaging Networks)?
This is used quite often in SoTA quantization algorithms, which definitely count as production
It is like Monte Carlo integration and the curse of dimensionality. Yes, you can use this for basically anything but it pays off only if the object you are trying to integrate is multidimensional, otherwise traditional methods outperforms
Isn't the 'we need gradients' a foregone conclusion?
Gradients can be calculated numerically, meaning that any method that samples the cost function and makes optimization decisions based on that can actually compute gradients if it needs to.
Random selection seems to favor post training
https://arxiv.org/pdf/2603.12228
If it's cheaper then I guess it will might get practical. Also more likely that mind uses simpler techniques more similar to these.
I wonder though if someone tested mutating training objective though, like keeping original loss/goal and somehow defining loss differently and then comparing against original. This intuitively feels like how mind tries to handle difficult tasks.
"A strict complexity gap between gradient-based and derivative-free Lipschitz convex optimization" has been known for decades. It's covered in standard textbooks like Nesterov's "Introductory lectures on convex optimization". That AI result is about establishing the gap in a fairly niche accuracy regime.
Thanks for the correction, I had a feeling that should be true but the headline was at top of search results and I'm not close enough to the field to know off the top of my head.
> Derivative-free optimization can be useful for genuinely discontinuous objectives [1], but common neural network objectives are smooth and/or Lipschitz.
There is even an analog to the continuous derivative for discrete binary functions, called "Boolean variation": https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...
Like for derivatives, there is a chain rule for Boolean variations, so you can use something like backpropagation, but without needing any expensive floating point math. Though I don't think this has been used much so far. There must be some other downside.
And yet the best learning algorithm we have is derivative-free!
Yes, SVMs and Random Forests still rule the many worlds of classification problems often not in the spotlight.
3 replies →
Perhaps the optimizer is worse but the architecture or objective function is better.
Imagine how good it would be with derivatives
Bayes Optimal Classifier?
I'm skeptical as to whether zeroth-order methods really lend themselves to a Bitter Lesson argument. First-order methods don't explore the loss landscape optimally, but the loss function tends to be nonconvex, and zeroth-order methods don't address that issue head on. Dust smooths, and so do applicable first-order methods. Remove the nonconvexity issue, and I suspect Dust's purported advantages evaporate (based on published theoretical work), so it's pretty odd to me that the paper never discusses convexity.
I could buy that this method scales better than previous zeroth-order methods, and that's interesting, but it doesn't seem like enough of a moat to keep improved first-order methods from drinking its milkshake, except in cases where a zeroth-order method is already a primary option: the network needs to call a simulator that doesn't expose gradient-like information. (In cases where gradients don't exist, I'd still argue for other options, e.g., Clarke-generalized gradients where applicable, so long as those can be computed with the available information. I know this technology has been published for automatic differentiation, so I would imagine it could be incorporated into backprop and used with a suitable optimization algorithm.)
Is it fair to define Reinforcement Learning (RL) as forcing a gradient onto a system / simulator?
Though, I suppose RL has non-gradient based methods too.
Orders-of-magnitude improvements in compute efficiency are needed to become a practical replacement for backprop… but those improvements are coming.
As a mixture, could activation-space search produce useful teaching targets for backprop?
Zeroth-order search would discover candidates, first-order learning would consolidate them. The potentially valuable step is converting a sparse judgment into a reusable training target.
This also changes the relevance of convexity.
I'm not sure I follow why your argument changes the relevance of convexity? If the problems were convex, I don't think we'd be having this discussion -- first-order methods would tend to win.
I'm excited to see a new thing in the Zero-Order Optimization (ZOO) world. I see most ZOO methods not as a "replacement for backprop" the way they are often marketed, but as a technique that can potentially work well in regimes where backprop is fundamentally weak. One of my favorite papers [0] in recent memory, for instance, is about using central-difference random gradient estimation (CD-RGE) to train large RNNs without using backprop-through-time. They also show it works decently for hard-to-optimize differentiable-neural-computers. Since reading that paper, I've had a lot of fun trying to apply this technique to settings I would describe as "a traditional differentiable neural network being applied in a non-traditional environment where you can't just call loss.backward and hope for the best". I am excited to try Dust out on my personal project now :)
> There are many interesting open questions. The first is whether, and how, Dust can find better directions than backprop’s first-order gradient
Both algorithms are bound by the same Pareto frontier based on the Empirical Risk Minimisation Principle, so they’re already on the same trajectory. Interestingly backprop is limited by conditioning of the Hessian matrix in order to converge (differentiate correctly). So removing this limitation is actually a great step. I’m excited to see a comeback of evolutionary methods because they’re much more general, albeit costly and naive. We’re now very close to what can be described best as brute forcing the Pareto frontier out of our datasets. Not sure that’s what we want but I have no better ideas either.
> Interestingly backprop is limited by conditioning of the Hessian matrix in order to converge (differentiate correctly)
What does this mean?
Pareto frontier the Pareto frontier and then with a little Pareto frontier you might get to the Pareto frontier
P^2
5 replies →
Even though this is way more expensive than backprop, could a hybrid approach where you fine tune an existing checkpoint that's been backpropped unlock further gains? It would be cool to apply this to different stages and see if that affects the learning trajectory
Intereting. There was a completely different take on how to "do" back propagation using optics at YC papers video the other day(I can't find the video). The technique presented diffusion networks with optics. If a hardware technique to make computation more efficient emerges, a lot of methods like this could apply no?
I would like to see wall clock time, energy, peak memory and downstream quality compared at equal loss. Until the effiency gap closes, this is an interesting research direction.
For me credit assignment without a global clock is extremely important, i believe it's a necessary step towards end to end training of models In-materio.
It sounds like this is less computationally efficient than backprop, but more easily parallelizable. Is that fair?
Not necessarily, backprop is highly parallelizable since it is just a bunch of matrix mults.
Something like Dust skips the backward pass on backprop. But other techniques like Neural Predictive Coding can be completely asynchronous, each "weight" can fire independent of those far away from it. Innocenti, et. al have shown that NPC gradients converge to backprop within a certain "regime".
The win with asynchronous techniques like NPC is that you do not need the extreme co-ordination that backprop requires and hence should be computationally much easier given the right device.
Although at this point the industry has so much money in the forward-backward pass system that I doubt a backprop successor would win unless someone makes NPC hardware feasible and can prove scaling up to billions of params
Also has the "advantage" of being slightly more biologically plausible as the optimization happens locally rather than globally.
That idea was taken further by N'dri et al in PCL, in which "activation energy" was minimized as well, and inhibitory neurons added https://www.nature.com/articles/s41467-025-64234-z.pdf
While trying to find the link for that I stumbled upon
https://arxiv.org/pdf/2605.12732
Which also looks pretty interesting
1 reply →
The reason why I don't see the promise for ML-only applications is that the coordination backprop requires comes very cheap to us.
"Much easier given the right device" - the "right" there just isn't shaped like the devices we actually build. And the price of "not having backprop" is usually expending more FLOPs, getting worse sample efficiency, etc.
The biggest "device" that doesn't do backprop is the brain, and that's because the brain doesn't have the connectivity or the coordination to pull it off. Both of those are "expensive" for something like it to implement. Cheap for us though. We aren't stuck with neurons that only get locally available information and have to implement learning rules based on that. So, skill issue?
6 replies →
Knowing nothing about this, I wonder if it could be useful in situations where we can’t reliably sync with all the workers. Something like folding@home, where all the workers are just shaking weights and if one of them finds a winner it uploads to the central server?
4 replies →
I think Jeff Dean is right in that we will see much more specialised silicon in the future.
If something more bio inspired ie. predictive coding and in-memory compute fundamentally makes continual learning and much lower energy consumption possible there will be specialised hardware for it at some point
FWIW I think the brain has multiple “learning rules” and operates at multiple timescales
Would these alternatives to backprop make it more feasible to have constant live-training going on in a model? Giving it something akin to neuro-plasticity?
3 replies →
At massive scales 0th order methods will parallelize better than backprop especially along depth, u can train very deep models pipeline parallel without bubbles
It's computationally expensive and infeasible but what's the upside?
Genuine question due to unfamiliarity with the subject.
I think it can be useful for optical hardware
Bigger Dust models needing fewer population samples?
This is super yawn-worthy. Instead of backprop for the exact gradient you can run forward passes a thousand times with perturbed weights and get a Monte Carlo estimate of the gradient. Not very clever. Extremely NOT useful.
But I guess the industry is littered with techniques for computing the same thing but vastly slower that some people find interesting. Homomorphic encryption. Zero knowledge proofs. Blockchain computing. Except in those cases there might be a legitimate reason to use it occasionally.
[flagged]
Moving beyond traditional backpropagation for transformer pretraining opens up fascinating avenues for alternative learning dynamics and architectural efficiency.
- stupid question from a neural network rookie
- isnt the whole point of back propoagation so that you dont guess weights by brute forcing them since that is computationally infeasible once you go beyond a dozen weights?
- if you dont use backpropogation, how exactly are the initial weights assigned them if they are not random values?
Hmm I was more interested in this alternative to backprop:
https://news.ycombinator.com/item?id=49701182
[flagged]
[flagged]
[flagged]
Definitely more than meets the eye.
Remember kids, whoever is pitching space searching via complete enumeration just wants your wallet.
Dust's 243M model beating a 120x smaller one at most population sizes is the surprising part; bigger nets got more population-efficient, not less.