Comment by jnwatson
7 hours ago
I can't get Qwen 3.8 27B to do a simple code review on a fairly basic Python file. With thinking on it just ruminates forever and with thinking off it gives obviously bad borderline hallucinating advice.
Edit: I tried again with the 2.4T model and it still ruminates to death, but with thinking turned off, it generated genuinely useful advice.
Edit2: adding
--reasoning-budget 8000 --reasoning-budget-message "Reasoning budget exhausted; give the final answer now." --reasoning-effort low"
to the llama.cpp executable parameters produces pretty good output.
One of the big learnings from 3.8 27b is adding reasoning budget really hurts the model. you need to let it spin for as many thinking tokens as it wants to to get it out. Another big takeaway is reasoning effort set to low doesn't save you tokens: low is pretty uncertain about things so it ends up thinking more (you can find some tests from folks on youtube). The final question, as always, is what quant are you running it at? KLD matters _a lot_ when it comes to its performance and it especially manifests with MTP/DFlash acceptance rate which makes those long thinking traces take a long time.
It literally ran forever without a reasoning budget. I tried even the 2T model and cut it off after a half hour. This is to review a few hundred-line source file. It was consistent behavior from 2T to vanilla 27B to my ablated distilled version.
how are you running the 2.4t model locally if you don't mind me asking
Oh I didn't; only a distilled 27B model ran locally. This was just to figure out if distilling caused the problem.