Comment by aftbit
1 day ago
>In keeping with our commitment to user responsibility, following the official launch of V4.1 Flash and prior to the release of V4.1 Pro, all requests to the Pro model will be routed to V4.1 Flash and billed at Flash's price
Please don't do this kind of thing. If a user has validated a workflow on V4 Pro, they might not want to suddenly start testing it in production on V4.1 Flash. Instead, keep V4 Pro around but deprecated for a defined period of time, then remove it.
At least as open weights models, it's possible to use something like Together.ai or OpenRouter to run the V4 Pro model as long as other providers keep it up.
Usually I would very much agree with you, but those things are not deterministic so if that's an issue for you you're probably not making the right choices.
That's a narrow take. Non-deterministic doesn't mean random; workflows can be reasonably validated and consistent to some known degree.
I work for an education department that serves a chatbot for students, and model changes go through painstaking content safety reviews. I initially assumed it's just a bunch of bureaucratic paranoia. But every other model upgrade has a measurably different adherence to the existing system prompts about not talking to the kids about sex and drugs and mental health issues.
I work in the same field for one of my company, in europe, and if you're not self hosting sorry but your worries are not something I can accept because models are very much not reliable on that front, let alone when you let the host decide HOW to serve a model (ressources allocated, different version of the same model, etc ...).
I'm not being a d**, just saying, the problem you have is something that I have faced EXACTLY, and at least here it's not working until you host in house or remote but on raw hardware. Otherwise it keeps having subtle changes, and you will notice no LLM API providers has guarantees about these.
3 replies →
The user cares about the distribution of outputs. That distribution is structurally determined by the distribution of inputs (i.e. prompts), the weights of the model, and (these days) the dynamics of the harness guiding successive generations.
The only way to characterize whether a choice is 'right' is to characterize the output distribution (i.e. evals)! Changing the underlying weights necessarily invalidates whatever characterization may have been done. One may assert that one's harness regularizes outputs back toward the desirable distribution, or one may hope the different weights induce a sufficiently similar output distribution.
But no, one should not be completely agnostic to the choice of weights just because there's some nondeterminism.
They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper.
Just the risk of such a thing means regression testing every time you update the model, and you want to be able to run that testing on your schedule rather than having it forced on you.
> They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper.
This isn't true. Even Sol messes up JSON formatting for me on occasion.
Do not delude yourself into thinking these things are reliable. They are not.
5 replies →
In deterministic scenarios, this kind of switch can sometimes actually be safe, because we’ve used various engineering techniques to converge from non-deterministic behavior to deterministic decisions. On the other hand, in scenarios that are inherently non-deterministic, the impact of such a change is much harder to predict, so we need comprehensive evaluations to assess the extent of its impact.
Nope, strong disagree. The model is one small part of the process harness; behaviors are usually routable with expected propensities. Unexpected model changes avoiding change management messes with monitoring and observability thresholds. Stochastic controls are a real thing when you have your distributions defined; your workflows on a new model will throw that expected prior out the window.
I think it depends on the workload, which put you in the right (in some case, it's dangerous to do that), but for me even for very clean always correct path I always assume a model can, at time, have a vector brain fart (because it keeps happening).
Also in principle it's similar to Anthropic downgrading.
Personally I use the basis that if I don't self host (I include remote host, but that I pay per hosting nor per model or api), it can change behavior without me asking. But they shouldn't, but it doesn't matter that's what they do.
Crossing the street and Russian roulette both have non-deterministic risks of injury. And yet I would be bothered to find out that that on my way to work, I was playing Russian roulette by surprise.
Yes and if you decide to play not using a game rulebook but a website that call it "game A" you can't be shocked if "game A" switched from one to the other, even though the doc said opposite yesterday.
2 replies →
A dice roll is non-deterministic.
Replacing a six-sided die for an eight-sided die also keeps rolls non-deterministic.
That doesn't mean it's fine to just replace the dice mid-game.
Sure, but they are replacing same generation model to another not just switching opus to gpt.
1 reply →
These models have specific behavioral characteristics trained into them from reinforcement learning and prompts optimized for one aren't guaranteed to transfer to the new generation. Think if the difference between gpt 5.4 and 5.5 and then 5.5 to 5.6 for example. 5.5 was "better" than 5.4 for struggled more across compaction boundaries and needed much more precise instructions before 5.6 sol recovered some of 5.4's ergonomics. All from the same lab but each model was trained with specific behavioral patterns that were basically product decisions. I would be quite annoyed to find that a model provider was routing a promt optimized for one model to a different one, especially for a dumber/cheaper non frontier model that's not going to be as good at just figuring out what you meant
It’s like being told one day that one of your teammates will be replaced tomorrow by another teammate who’s more capable (has a higher test score), regardless of how long you’ve already worked together and gotten used to each other.
Determinism was an explicit goal of DeepSeek-V4. From their paper: https://arxiv.org/html/2606.19348v1#S3.SS3
Of course, providers may not implement deterministic inference for various reasons, but it is possible.
Most people are not even referring to CUDA batching nuances.
They think that sampling is an inherent part of Transformers.
Even on this site, it is regurgitated with confidence.
Models are not deterministic, but they do have a flavor. When that flavor changes it can change the nature of output in a way that is undesirable.
Sort of like shooting a rifle - where the bullets hit is (to some order of magnitude, no philosophizing please) non-deterministic, but different very similar rifles will group differently and need to be appropriately adjusted to hit anything.
Yeah but then it doesn't matter if it's "the model changed to another one" or "the model changed but it's the same name". Point is, you're not hosting it, as far as you know it can change at any moment, build around that idea. Is that great no, is that ideal no, that's why I self host (I include actually renting online the capacity and hosting the model myself on it).
3 replies →
This is a great way to think of it, idiotsecant.
That flavor profile is known -- it's typical behavioral distribution is somewhat understood (and, often, common failure modes addressed). If JSON breaks about 20% of the time, and that drops for 2% or blows up to 90%, it can drive all sorts of issues (not the least, costs for retries).
I think keeping models around for a defined period of time is fine, but fracturing your model offerings like that (keeping around multiple versions of the same model) is very hard to do economically. The economics of the AI ecosystem are dominated by queueing theory constraints that make it extremely cheap to serve predictable traffic loads, and extremely expensive to serve unpredictable loads, and any time you split your offerings like that, you make both less predictable and therefore more expensive to serve both versions.
If I were paying anthropic prices, I'd expect it, but Deepseek is a super scrappy upstart in comparison and intentionally arbitraging on price. I would never expect them to do that.
I imagine they need the compute. Can expand market share with more users for same amount of compute.
But I agree with you. I have a dumb workflow that worked well with v4-flash-0731 and I suspect is directing to a newer model that now breaks it.
4.1 releases tomorrow, right now you're supposed to be served by same old model
I'm not sure that anyone is running production workloads against an API that bills twice as much for a chunk of the day. One of the best things about Deepseek is that you can host it yourself and get a ridiculous multiple of usage for what the same dollar amount would yield from their API
> I'm not sure that anyone is running production workloads against an API that bills twice as much for a chunk of the day.
I'm not sure that anyone will mind running production workloads against an API that bills half as much for a chunk of the day.
It'd be very expensive to get a setup that can run non-flash well.
If they were still at original price I'd get a couple of DGX Sparks myself to run Flash models at a decent quant/context combo.
LLMs add enough nondeterminism to a workflow. Swapping them without the user knowing adds substantially more.
relying on cloud models for anything that gives you ROI is tying a loose noose.
This is true :P.
I don't really agree, but we shouldn't have to debate it. An Auto option at each level would preclude this kind of decisioning. Pick a discrete model, that's what you get.
Pick Auto (Deepseek v4 Flash Auto vs Deepseek v4.x Flash), and let the vendor decide. I think OpenRouter uses this method.
In this case, Deepseek organization is under a lot of pressure due to compute constraints. It would be better if they just throw a 404 instead of rerouting though so customers are not surprised by subtle changes in behavior.
Yes exactly. Automatic model downgrade seems horrible for a lot of production workloads, even if you are deterministically constraining the behavior of your agents.
I wonder if this is a sign of things to come for dirt-cheap model hosting: no servers running old versions, only new versions. Just to keep costs down.
its probably more expensive to run, and I'm thinking 4.1 flash is a smaller more efficient model. You can always host your own. This is what they need to do to stay competitive.
llm is not a deterministic program. the same model won't even return deterministic answer. what's the point keep the model freezed?
if you want deterministic returns, you should set the temperature to 0 to get the best possibility of deterministic.
Strict determinism is a different, but related, issue.
E.g. if I've written a role playing character using a specific model I may want to pin the character to that model until I've been able to test the model being "better" doesn't affect the feel of the character before switching. That doesn't mean I need the character's responses to be completely deterministic, but that doesn't imply I'm fine with the character having a different quality or feel of response just because the new model is out.
It'd be nice if there was a more explicit way to signal in the request "I want what you think is best per dollar for this class of answer" vs "I want this model to answer".
They expect vibe coders to use their models only lol
You absolutely cannot consider an LLM production build number something to be pinned against as a static dependency in a product chain, so it's a non-issue.
Yes you can and you should. Providers have SLAs for when models roll off support and this has been the case for APIs long before LLMs. For example https://platform.claude.com/docs/en/about-claude/model-depre... and https://developers.openai.com/api/docs/deprecations
It's very easy to tell who is not running production applications using these models based on comments like this
Let me give you the benefit of the doubt, can you expand what you run in production?
2 replies →