Comment by Bolwin

Distillation requires you to have the actual logits of each token from the teacher model, which in practice means having the model itself.

What you're describing is just synthetic data.

Note Anthropic misused the term in their post about Chinese model distillation, deliberately I assume.