Comment by cyanydeez

distillation doesnt add anything; all it's doing is reconfiguring some root weights that get drowned out by noisy training and/or datset issues. It strengthens commonalities.

but there's no new information being created.