Comment by nolok
21 hours ago
Usually I would very much agree with you, but those things are not deterministic so if that's an issue for you you're probably not making the right choices.
21 hours ago
Usually I would very much agree with you, but those things are not deterministic so if that's an issue for you you're probably not making the right choices.
That's a narrow take. Non-deterministic doesn't mean random; workflows can be reasonably validated and consistent to some known degree.
I work for an education department that serves a chatbot for students, and model changes go through painstaking content safety reviews. I initially assumed it's just a bunch of bureaucratic paranoia. But every other model upgrade has a measurably different adherence to the existing system prompts about not talking to the kids about sex and drugs and mental health issues.
I work in the same field for one of my company, in europe, and if you're not self hosting sorry but your worries are not something I can accept because models are very much not reliable on that front, let alone when you let the host decide HOW to serve a model (ressources allocated, different version of the same model, etc ...).
I'm not being a d**, just saying, the problem you have is something that I have faced EXACTLY, and at least here it's not working until you host in house or remote but on raw hardware. Otherwise it keeps having subtle changes, and you will notice no LLM API providers has guarantees about these.
It's not that you're a d*, it's just that you lack any kind of nuance
There's a whole spectrum between self-hosting open weight models and having a cloud provider swap models from under you
Should you self host a model if want to maximize predictability to the limit? Yes. Does that mean it's wrong for someone hitting a model on API to expect that it won't switch to a completely different model under the hood from one day to another? Probably not.
1 reply →
If self-hosting is the only solution you can think of to maintain model stability, it's no wonder you missed the point. We already do all those things you mentioned, even with cloud providers. Model versioning exists for exactly this purpose, just like it does with any software dependency. We can specify sonnet:v1.23 and decide if and when to upgrade.
But even if we only asked for sonnet:latest, the last thing we'd expect is opus. Model names should be indicative of breaking changes, and change management doesn't just go in the bin because of non-deterministic tools.
The user cares about the distribution of outputs. That distribution is structurally determined by the distribution of inputs (i.e. prompts), the weights of the model, and (these days) the dynamics of the harness guiding successive generations.
The only way to characterize whether a choice is 'right' is to characterize the output distribution (i.e. evals)! Changing the underlying weights necessarily invalidates whatever characterization may have been done. One may assert that one's harness regularizes outputs back toward the desirable distribution, or one may hope the different weights induce a sufficiently similar output distribution.
But no, one should not be completely agnostic to the choice of weights just because there's some nondeterminism.
They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper.
Just the risk of such a thing means regression testing every time you update the model, and you want to be able to run that testing on your schedule rather than having it forced on you.
> They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper.
This isn't true. Even Sol messes up JSON formatting for me on occasion.
Do not delude yourself into thinking these things are reliable. They are not.
Is nobody using structured outputs? They use constrained decoding at the generation stage to ensure the probability of tokens that would break the format are set to 0. I kinda figured everyone was doing this at this point.
4 replies →
In deterministic scenarios, this kind of switch can sometimes actually be safe, because we’ve used various engineering techniques to converge from non-deterministic behavior to deterministic decisions. On the other hand, in scenarios that are inherently non-deterministic, the impact of such a change is much harder to predict, so we need comprehensive evaluations to assess the extent of its impact.
Nope, strong disagree. The model is one small part of the process harness; behaviors are usually routable with expected propensities. Unexpected model changes avoiding change management messes with monitoring and observability thresholds. Stochastic controls are a real thing when you have your distributions defined; your workflows on a new model will throw that expected prior out the window.
I think it depends on the workload, which put you in the right (in some case, it's dangerous to do that), but for me even for very clean always correct path I always assume a model can, at time, have a vector brain fart (because it keeps happening).
Also in principle it's similar to Anthropic downgrading.
Personally I use the basis that if I don't self host (I include remote host, but that I pay per hosting nor per model or api), it can change behavior without me asking. But they shouldn't, but it doesn't matter that's what they do.
Crossing the street and Russian roulette both have non-deterministic risks of injury. And yet I would be bothered to find out that that on my way to work, I was playing Russian roulette by surprise.
Yes and if you decide to play not using a game rulebook but a website that call it "game A" you can't be shocked if "game A" switched from one to the other, even though the doc said opposite yesterday.
That would absolutely be a dick move on the part of the website and would confuse users.
Also, in this case, the game name is not “Game A” but something like “Deep Seek v4 Pro”, which they have previously chosen to use to describe Deep Seek v4 Pro, not Deep Seek v4.1 Flash.
You should look at why you are protecting your earlier comment rather than changing your mind or understanding why so many people disagree with you.
A dice roll is non-deterministic.
Replacing a six-sided die for an eight-sided die also keeps rolls non-deterministic.
That doesn't mean it's fine to just replace the dice mid-game.
Sure, but they are replacing same generation model to another not just switching opus to gpt.
But it’s more like them swapping Opus 5 for Sonnet 5.5 or something. IMO OP’s right to argue against it.
These models have specific behavioral characteristics trained into them from reinforcement learning and prompts optimized for one aren't guaranteed to transfer to the new generation. Think if the difference between gpt 5.4 and 5.5 and then 5.5 to 5.6 for example. 5.5 was "better" than 5.4 for struggled more across compaction boundaries and needed much more precise instructions before 5.6 sol recovered some of 5.4's ergonomics. All from the same lab but each model was trained with specific behavioral patterns that were basically product decisions. I would be quite annoyed to find that a model provider was routing a promt optimized for one model to a different one, especially for a dumber/cheaper non frontier model that's not going to be as good at just figuring out what you meant
It’s like being told one day that one of your teammates will be replaced tomorrow by another teammate who’s more capable (has a higher test score), regardless of how long you’ve already worked together and gotten used to each other.
Determinism was an explicit goal of DeepSeek-V4. From their paper: https://arxiv.org/html/2606.19348v1#S3.SS3
Of course, providers may not implement deterministic inference for various reasons, but it is possible.
Most people are not even referring to CUDA batching nuances.
They think that sampling is an inherent part of Transformers.
Even on this site, it is regurgitated with confidence.
Models are not deterministic, but they do have a flavor. When that flavor changes it can change the nature of output in a way that is undesirable.
Sort of like shooting a rifle - where the bullets hit is (to some order of magnitude, no philosophizing please) non-deterministic, but different very similar rifles will group differently and need to be appropriately adjusted to hit anything.
Yeah but then it doesn't matter if it's "the model changed to another one" or "the model changed but it's the same name". Point is, you're not hosting it, as far as you know it can change at any moment, build around that idea. Is that great no, is that ideal no, that's why I self host (I include actually renting online the capacity and hosting the model myself on it).
You've added a similar response like 5 times, but I don't see you adding more information.
Yes - model hosts can do nasty things to you aside from changing the underlying model. That doesn't mean it's cool to have them change the model automatically.
Yes, it would be preferable to have complete control over your model serving, and no - not everyone is in a position to do that themselves.
1 reply →
Often via api you can pin to the specific snapshot. Yes the model provider can screw you over - it's physically possible for many vendors to screw you over, that doesn't mean that's not a dick move and that you can't expect/push for better behavior
This is a great way to think of it, idiotsecant.
That flavor profile is known -- it's typical behavioral distribution is somewhat understood (and, often, common failure modes addressed). If JSON breaks about 20% of the time, and that drops for 2% or blows up to 90%, it can drive all sorts of issues (not the least, costs for retries).