Comment by sspiff

They run one step/iteration on an additional chunk of training data, then use the snapshot of the weights after that iteration in a separate validation benchmark while continuing to train on another chunk of data for the next iteration.

They result of the benchmark does not feed back into the training, it simply serves to provide a measurement of progression over time.