Comment by slashdave
2 years ago
Mini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an important feature of deep learning). This is deliberate and necessary to explore the gigantic parameter space of these huge LLM models.
Sure, but one might think that training on self-generated data would keep you in a constrained subset of minima, but that is not the case.