Comment by DoctorOetker
6 hours ago
Nothing prevents interleaving inference and gradient descent / RMAD / backpropagation from being interleaved.
Is a man a father or a son? It's a false dilemma, it can be both.
Is a training step pretraining or finetuning? It's the same mathematical operation, with the only difference the intention or reason of applying the training step.
LoRA throws a tiny amount of sand in these gears, but I generally agree there is near zero difference beyond the semantics we mere mortals assign to written tokens