Comment by bigwheels
12 hours ago
Does this suggest Mistral is distilling Deepseek et. al.? Can distillation keep passing through with high fidelity, or is it lossy like copying a VHS tape?
12 hours ago
Does this suggest Mistral is distilling Deepseek et. al.? Can distillation keep passing through with high fidelity, or is it lossy like copying a VHS tape?
It doesn't, even remotely. It's a typical failure mode in all models, some do this more often. It's not related to samplers either. Not necessarily Chinese characters either, they likely just occupy a large portion of vocabularies. For example Gemini 2.0 used to glitch in Telugu and Ukrainian, Claude 2.x and 3.0 used to switch between random languages for a few words when writing in non-English, once in a blue moon.
No. I've had pretty much every model I've worked with (list too long to recite) spit out the odd Chinese, Korean, Thai or other Asian chunk of letters from time to time. ChatGPT in various incarnations has suddenly switched to Chinese in mid-output and remained that way until told to switch back. The only one I haven't really seen this from as far as I can recall is Gemini, but then again, I haven't used it as much because it was performing quite poor for what I was doing.
I interpret it to mean there is a lot of Chinese in the training corpus.