Comment by dakolli
9 hours ago
That's because Deepseek invented the paradigm of prompt caching, they are the SOTA when it comes these techniques. Despite them open sourcing all their research, nobody beats them.
edit: I do wish openrouter would let you sort providers by Cache Hit % and Cache cost. These are the only things that matter to me at this point when choosing a provider.
Cache hit % on openrouter is not a good metric, it's mainly driven by openrouter's own provider juggling than the providers themselves
OpenRouter randomizes which provider gets your request by default right? I think you have to pass a specific provider in the request to prevent that. (Or set up a preset or something.)
This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.
> OpenRouter randomizes which provider gets your request by default right?
I'm not sure it's wholey accurate to say they "randomize" the provider, rather my assumption based on usage is that it's something like cheapest-ish/responded to the request within some reasonable-ish time/etc algorithm that chooses the provider on each request - which seems, remarkably questionable in terms of optimizing for user experience or hidden user costs.
> This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.
I so very much recommend this approach. My avenues that automate llm calls to openrouter are setup to make api reqs to openrouter to determine best price/response/etc and then pin the request to that (and, preferably, a fallback if there's reasonable difference between #1 and #2) provider for that session. Otherwise you're going to have a bad time.
I'd imagine this could make things interesting in cases where one provider is offering different quants than the others and openrouter is just swapping you back and forth on a long agentic session.
> This behavior makes it so you don't benefit much from the caching
I don't believe this is correct? AFAIK once it routes you to a provider for a given conversation that choice is sticky unless you hit technical difficulties. (It's more complicated than that, they recently added named routing strategies that you can append to the model name.)
IMO the relevant metric is cache TTL which isn't typically published AFAIK.
This is not true, there isn't even a way to see a cache hit % model for a specific model, that wouldn't make any sense. You are confusing what I'm saying with cache cost, that has nothing to do with effective cache hit %. I'm talking about when you click on a specific provider for a specific model, you can scroll down on the view and see their cache hit % for that model [0].
These cache Hit % are accurate, I've done a ton of testing of this myself. The cache hit % is one of the most important metrics as far as estimating cost. There are many providers with cheap cache reads, but have an effective cache hit % of 30%, making their cheaper cache pricing meaningless compared to another provider who charges more but has a 85% cache hit percentage.
[0]: https://openrouter.ai/deepseek/deepseek-v4-flash-0731?endpoi...
scroll down on the provider/model card and you'll see a field called cache hit %, its different for every provider/model.
I don't use routing on openrouter, I strictly use models with a single provider and no fallback, at least for use with harnesses its pretty dumb to route requests to multiple providers you are busting your cache every other request and increasing costs by 20-50%.
I think you're arguing the same general point that the person you're responding to is. But you're saying he's not understanding - he understands that they report a cache hit % but you can't look at that public metric with any level of accuracy _because_ most people aren't pinning their providers and they _are_ getting juggled around which is bringing that metric down. That's not to say that specific providers might have issues or worse cache implementations - but it stands that if openrouter is juggling the requests back and forth by default then _that alone_ is breaking caches on those requests in huge numbers.
1 reply →
> Cache hit %
I thought you had to actively manage caches, do you not?
>Deepseek invented the paradigm of prompt caching
Caching was always here, you don't need to do anything special to get it on a single user local backend running a base model or a chatbot in the first place. Among commercial providers, OpenAI adopted it in 4o first.
You can click the table headers to sort Ascending/Descending.
You can sort by the cost cache cose, but you cannot by cache hit %. You have to click on the provider and see what their cache hit % is. A provider could have a super low cache cost, but a 50% cache hit percentage, making the cheap price of cache read's meaningless.
Wait, what does that number mean? I thought it always uses the cache price when the prefix matches.
1 reply →