Comment by bigwheels
3 hours ago
Does this suggest Mistral is distilling Deepseek et. al.? Can distillation keep passing through with high fidelity, or is it lossy like copying a VHS tape?
3 hours ago
Does this suggest Mistral is distilling Deepseek et. al.? Can distillation keep passing through with high fidelity, or is it lossy like copying a VHS tape?
It doesn't, even remotely. It's a typical failure mode in all models, some do this more often. It's not related to samplers either. Not necessarily Chinese characters either, they likely just occupy a large portion of vocabularies. For example Gemini 2.0 used to glitch in Telugu and Ukrainian, Claude 2.x and 3.0 used to switch between random languages for a few words when writing in non-English, once in a blue moon.
I interpret it to mean there is a lot of Chinese in the training corpus.