Comment by martinald
10 hours ago
I wrote about this a couple of weeks ago. It's actually often the biggest cost and it tends to be hidden away on most platforms!
https://martinalderson.com/posts/watch-out-for-cache-read-co...
Btw I still haven't came across any decent model that is <$0.01/MTok cache costs apart from deepseek thru their official API (even with the price increases).
Seems like a bit of an opportunity for someone to take - drop cache read costs significantly.
For me, an average long session results in about 200-300M cached input, 4-800K input, 2-400K output. Mostly the lower bound. Output depends on how much the model thinks.
There are two problems here:
- cache hit pricing (both Muse Spark 1.2 Contributor and MiMo 2.5 are around the $0.002-3/M mark)
- cache persistence time
Muse Spark drops the cache in less than 5m. MiMo keeps it around for at least an hour based on my experience with whoever is serving it for OpenCode. This difference itself will inflate bills massively.
A 500K token input repeatedly read by MS 1.2 for full input price 12 times an hour = $0.60. You would be expecting $0.012. So a 50x difference. Same thing on MiMo 2.5 is $0.018 because of longer cache times.
That's because Deepseek invented the paradigm of prompt caching, they are the SOTA when it comes these techniques. Despite them open sourcing all their research, nobody beats them.
edit: I do wish openrouter would let you sort providers by Cache Hit % and Cache cost. These are the only things that matter to me at this point when choosing a provider.
Cache hit % on openrouter is not a good metric, it's mainly driven by openrouter's own provider juggling than the providers themselves
OpenRouter randomizes which provider gets your request by default right? I think you have to pass a specific provider in the request to prevent that. (Or set up a preset or something.)
This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.
2 replies →
This is not true, there isn't even a way to see a cache hit % model for a specific model, that wouldn't make any sense. You are confusing what I'm saying with cache cost, that has nothing to do with effective cache hit %. I'm talking about when you click on a specific provider for a specific model, you can scroll down on the view and see their cache hit % for that model [0].
These cache Hit % are accurate, I've done a ton of testing of this myself. The cache hit % is one of the most important metrics as far as estimating cost. There are many providers with cheap cache reads, but have an effective cache hit % of 30%, making their cheaper cache pricing meaningless compared to another provider who charges more but has a 85% cache hit percentage.
[0]: https://openrouter.ai/deepseek/deepseek-v4-flash-0731?endpoi...
scroll down on the provider/model card and you'll see a field called cache hit %, its different for every provider/model.
I don't use routing on openrouter, I strictly use models with a single provider and no fallback, at least for use with harnesses its pretty dumb to route requests to multiple providers you are busting your cache every other request and increasing costs by 20-50%.
2 replies →
> Cache hit %
I thought you had to actively manage caches, do you not?
>Deepseek invented the paradigm of prompt caching
Caching was always here, you don't need to do anything special to get it on a single user local backend running a base model or a chatbot in the first place. Among commercial providers, OpenAI adopted it in 4o first.
You can click the table headers to sort Ascending/Descending.
You can sort by the cost cache cose, but you cannot by cache hit %. You have to click on the provider and see what their cache hit % is. A provider could have a super low cache cost, but a 50% cache hit percentage, making the cheap price of cache read's meaningless.
2 replies →