Comment by mike_hearn

You can do that with backprop too. Nothing says you can't inference on a set of weights at the same time as you produce an updated copy from them.

They say in the paper what it's about: mostly just scientific curiousity but such approaches might be useful for making DNNs more energy efficient via neuromorphic hardware in future.

For continual learning at the weight level there's the business model issue. The labs are already deep in the red, the last thing they want is to give up shared weights. The I/O and storage costs of that would make it infeasible. Already KV caches are a sort of dynamic 'fast weights' and those are expensive!