Comment by wolttam
4 hours ago
If anything I am confused by how high cache costs still are across many models
The really expensive part of inference - the compute and memory traffic required to perform prompt processing and decode - is paid for roughly once per token in a multi-turn session when prompt caching is working properly.
The difference to the provider's costs between:
- Submitting 100 back-to-back requests, each adding 1k tokens to the context
- Submitting 1 100k token context request
Is negligible. Yet the consumer is going to pay a lot more in the former case for all those repeatedly submitted input tokens that you get charged the cached rate for (but which required no additional compute on the provider side).
DeepSeek V4.1 has an insanely optimized KV-cache that takes around 900MB per million tokens, making it feasible to move to disk between requests. Makes sense they're cheap
The amazing part of DeepSeek's caching is that the TTL is (although not guaranteed) like 8-24 hours. Makes a big difference to how much time I can spend reviewing code, when most everyone else offers maybe 1 hour (for subscribers - often minutes for API users).
I mean, memory prices are high. They ain't getting cheaper. And KVCache is held in memory. The more people are using cache, the more memory you need. It's not free to cache it.
Plus there's complicated cluster-wide KVCache distribution schemes, built around RDMA networking, etc etc.
By far the most expensive hardware cost now of inference is the memory cost. KVCache is still using it. Though obv GPU compute is using more energy
It's not just like... "stick it in Redis, all done!"