Comment by Tuna-Fish

> You can do that with backprop too. Nothing says you can't inference on a set of weights at the same time as you produce an updated copy from them.

Nothing but economics.

Inference requires dramatically more memory, meaning it gets radically more expensive, if you are doing backprop alongside. This is why it is not done, despite the obvious advantages. If this doesn't require storing as much data for updates, it might make such systems more feasible.