Comment by gunalx
7 days ago
The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.
7 days ago
The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.
Oh, interesting. "!" is token id 0 in the Qwen tokenizer; I wonder if there's some tokenizer shenanigans either in inference or training that end up causing this specific behavior.
Wait in a regular sentence, what is the probability of "!!!" being followed by "!"?
Sounds like the model is not following a proper probabilistic choice here, so maybe more a programming error than a model training error.
After the third !, the probability of a fourth probably skyrockets =)