Comment by Der_Einzige
2 days ago
Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings.
If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.
2 days ago
Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings.
If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.
For those us us who don't know, what do those parameters do and why are they better?
Temperature, top-up, top-k, min-p all control which token the model predicts next and how likely it is to select one token over the other.
You might understand this as "The capital of France is..." and the model isn't always going to select "Paris". Sometimes it will start a descriptive sentence or even get the answer wrong.
That selection of the next token is what these settings control, and lots of sub-optimal selections compound over time to produce a junk response.
I broadly knew that about temperature, but lack the background in machine learning/statistics to differentiate top-n-sigma from top-k/top-p.
6 replies →
The Qwen team published the same sampler settings for 3.8 and presumably they used those while testing on benchmark. Do you believe they could have achieved higher result with top-n-sigma?
Diverging from the sampler used in RL training is not good for long multi-turn results-- it's a great way to knock models into reasoning loops that wouldn't otherwise.
Peer reviews NeurIPS caliber paper to prove that? Because I can show you one titled "Long context generation is a sampling problem"...
Show us, we're curious. Did you upload to ArXiv yet?
How is terrible settings a case of FUD?