Comment by txhwind

other questions:

- can it be used to relax timing order requirement in pipeline parallelism? Each node update lambda on communication, and optimize weights at other time. - given that BP is using SGD, can batches and T share the same timeline in optimization, while keeping the descent direction?