Comment by swe_dima
11 hours ago
It's scary relying on Google's models.
I have a very price sensitive workload that used to run on flash 2.5 lite - it's deprecated now.
The replacement 3.1 flash lite is a lot more expensive, but now also has a sunset date.
3.5 flash lite is even more expensive.
So the price is rising and you have no choice but to keep paying more and more.
I moved directly from 2.5 flash lite to deepseek v4 flash, its already cheaper and if your prompt caching is good you can save so much more money.
could you explain how to optimize prompt caching or point to a doc about it?
Anything Sam Rose is worth reading: https://ngrok.com/blog/prompt-caching
but the implementation will be up to your provider and harness, for deepseek, they expose some numbers: https://api-docs.deepseek.com/guides/kv_cache/ and Anthropic has a list of actions invalidating your cache: https://platform.claude.com/docs/en/build-with-claude/prompt...
Basically, you avoid anything dynamic: model change, tool change, etc it's also important that your system prompt or main prompt doesn't have non-static data like the date/time/place or someone's name (the person you interact with in a chatbot for example). That should be left to tool call or search.
6 replies →
Not an open source, but I discuss it in my book with examples for OpenAI/Anthropic/Gemini, https://crimede-coder.com/blogposts/2026/LLMsForMortals.
All of the models, you need to have a consistent input to get the cache hit. So if you are chatting with a document, and change the system prompt, it will be a cache miss, even if the rest of the items are all the same. If you even pass in the document in not the same order as the prompts, it will be a cache miss. Or if you add tool calls or structured outputs, it will be a cache miss. (Since those generally go at the beginning of the prompt call, not at the end.)
Most of the time when reading documents from URLs directly it will never cache. (Need to typically pass in the bytes directly, or use the provider document store index.)
Gemini has a 4096 minimum token size with the 3 version models before even getting a cache hit. OpenAI it is lower (1024), and is automatic, but only happens in increments of 124. Anthropic can also get cache hits at 1024 tokens, but you need to explicit ask for it (and pay extra).
Caching by default typically lives for 5 minutes since the last cache hit across providers. But some of them you can ask for longer. AWS for Anthropic models can be tricky with multiple endpoint routing, so can get cache misses if it happens to route to a different endpoint.
That’s part of why, since Firebase, I’ve tried to never depend on Google products for business, especially not GCP.
Features stay in Beta for ages, whatever that actually means, and released ones get deprecated things fast.
Where some of the competitions treats deprecating entire services as "let’s not put it on your frontpage, put deprecation notices all over the doc, and politely ask new users not to start new project with them".
Just switch the model, its not that much effort tbh. And u can also get a cheaper model than 2.5 lite for the same intelligence
its not always that simple. dropping in a new model is trivial, but highly specific workflows may rely on specific _invisible_ aspects of a model. when that model gets deprecated, the workflow needs to be rebuilt/re-tuned to work with a different model.
google's inability or unwillingness to provide stable timelines for model deprecation makes it risky to build complex workflows using their models
Load-bearing (whoops) quirks were noticeable months back, but haven't most flagship models become predictable and reliable?
1 reply →
100% agreed in the same boat right now. Feeling really screwed over by Google rn
You would be surprised how much of a difference the model makes for certain niche tasks.
For my use case, `gemini-3.1-flash-lite` is ~20% higher accuracy than the next best model of comparable cost (considering both proprietary and open-weight alternatives)
Well it is a bit surprising that 3.1 flash-lite could be better than deepseek-v4-pro (cheaper output and way cheaper cache so might cost less for quite a few use cases).
They are not anywhere close according to pretty much every benchmark (even v4-flash is considerably ahead and its way cheaper than flash-lite). Maybe tuning prompts/tools/etc. might be useful?
1 reply →
"Intelligence" being what, math? Coding? Unfortunately there's a billion use cases for LLMs whose performance is not at all captured by the popular benchmarks they're all trying to maxx.
if you are relying on a model for a business process, it should be simple enough to benchmark on that process
> So the price is rising and you have no choice but to keep paying more and more.
I presume you can't use deepseek?
There are plenty of 3rd party providers hosting deepseek models, if you don't want to use the 1st party API. 3rd party providers are generally slightly more expensive, but still quite cheap compared to other models of similar vintage and size.
sadly it's not multimodal
They know that there's big enterprises that will have a strong preference to work with another big enterprise instead of relying on a younger company. At least that's why I think they believe they can do this sort of thing and get away with it.
same here. our production workloads was on Gemini for 2 years. seeing Google unilaterally dropping perfectly fine models and charing you 50x more for worse results is not good.
we are switching to Deepseek.
Opencode Go is just the same. Each month I will I can do less. Dont ask me why?
I'm running price-sensitive data extraction workloads on flash 2.5 and its still the king when it comes to accuracy + cost, all the gemini 3 variants perform a bit worse and cost a lot more. Low-key freaking out, ngl
I felt the same way about openai's text-davinci-002 and code-davinci-002 (gpt-3.5). They were amazing completion models and openai basically dumped them with no equal cost or equal performance replacement. Instead all their models are opaque with no ability to work in completion mode where one actually controls the text input to the model.
These days no company even has completion models where one controls the text input fully. Worthless.
And somehow, the most annoying is not even the price hike, but it is that is you expect to build a product on any of theirs models, they spend their time being deprecated and you have like to be on the lookup to start from scratch selecting a model and fitting it every year or so... Impossible to have any stability...
All models are increasing in price. Everything up to now has been subsidized by investors, private and public.
same I just switched to OpenAI after using flash 2.5 lite for almost everything at our company. We spent thousands just to build this workflow now Google says screw off
>So the price is rising and you have no choice but to keep paying more and more.
You can also just write code like you did a year or two ago.
[flagged]
Can you give examples of other things they can do that would be worth paying for?
4 replies →
Mostly because the person I was replying to has commented about using it to write code.
If you're using it for other purposes, then I give you permission to ignore my comment; there's no reason to descend into name calling.
2 replies →
[dead]