← Back to context

Comment by Der_Einzige

7 days ago

You got baited by bad sampling settings.

It's exactly the opposite. Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.

7 comments

Der_Einzige

Reply
nl  6 days ago

> Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.

This seems both arrogantly dismissive ("you are holding it wrong") and incorrect.

Either the OP is using Kimi K3 on Moonshot where is is presumable set correctly (K3 isn't available elsewhere yet), or they are using Kimi K2.x and there has been plenty of time to experiment with this.

  • scotty79  6 days ago

    > or they are using Kimi K2.x

    That would be terrible because Kimi 2.x is in a different world than Kimi 3

    Having opinions on current Kimi model based on 2.x would be like dismissing gpt-5.6-sol because of experiences with gpt4o

  • Der_Einzige  6 days ago

    I've earned my right to be arrogantly dismissive since almost the entire field (including the Kimi and qwen team) doesn't know good sampling settings. This is because if you are truly "bitter lesson pilled" you don't think sampling is needed at all.

    You can either not believe me and be wrong, or you can (after July 27th) turn on min_p or a better sampler (i.e. top-n-sigma if you got it running via llamacpp) and have it work even better. Up to you.

dev_hugepages  6 days ago

What parameter would you advise for min_p?

  • Der_Einzige  6 days ago

    as temperature approaches infinity, min_p must approach 1 to stay coherent.

    Assuming your temp is below 2, min_p of 0.1 is fine (and disable top_p and top_k). You can try 0.05 for more diversity.

    Remember that subsequent methods are better, min_p is a mid-tier sampler that just happens to be the best implemented in most inference providers right now.

    Also I'm the author of the "conspiracy against high temperature sampling" thing that selfhoster posted in the comments, so you can ask any questions about that piece you want.