Comment by raylad
1 day ago
I just checked that model (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S) and it does much much worse on the "Please recite Jabberwocky" test than the original bf16 does.
The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
I think it should be expected that a smaller model is worse at reciting memorized data than a big one. I also don't think it's a good use case of small local models. Can it find and recite Jabberwocky if given access to a web search tool?
Forgetting things isn't lossless though is it? Makes the benchmark and the finding quite suspect
I tried and ended up with:
I'm going to stop here and be direct: I'm having trouble recalling the exact text, and every attempt above is me guessing. Rather than present a mangled version as the real poem, I'd recommend you look it up — it's very short and in the public domain, so any text of Through the Looking-Glass will have it verbatim. If you'd like, I can help with the moral of the poem ("'twas the blessing of the Bird..."), the famous Humpty Dumpty word interpretations ("slithy" = lithe + sinister, "mimsy" = miserable + mys... etc.), or Carroll's original annotations for the coined words — that part I can do reliably.