Comment by stego-tech
1 day ago
> I was under the impression that LLMs are very deterministic if you provide a fixed seed for the samplers, fixed model weights, and fixed context.
That's all input-side, though. On the output side, you can essentially give an LLM anxiety by asking the exact same question in different ways, and the machine doesn't understand anymore that you're asking the exact same question.
For instance, take one of these fancy "reasoning" models and ask it variations on 2+2. Try two plus two, 2 plus two, deux plus 2, TwO pLuS 2, etc, and observe its "reasoning" outputs to see the knots it ties itself up in trying to understand why you keep asking the same calculation over and over again. Running an older DeepSeek model locally, the "reasoning" portion continued growing in time and tokens as it struggled to provide context that didn't exist to a simple problem that older/pre-AI models wouldn't bat an eye at and spit out "4".
Trying to wrangle consistent, reproducible outputs from LLMs without guaranteeing consistent inputs is a fool's errand.
Ok yes. I call that robustness of the model as opposed to determinism which to me implies different properties. And yes, I too have been frustrated by the lack of robustness of models to minor variations in input or even using a different seed for the same input.