Comment by sudosysgen
7 hours ago
If you read DeepSeek's papers, you'll find a litany of architectural features that allow for a greatly reduced cache hit price by shrinking the size of the KV-cache.
7 hours ago
If you read DeepSeek's papers, you'll find a litany of architectural features that allow for a greatly reduced cache hit price by shrinking the size of the KV-cache.
How come no other big model seems to be able to deliver the same type of extremely low cache cost though, if their techniques are public?
I think the "architectural features" are part of the model, not the kv cache. So implementing it would be difficult and expensive.
Deepseek V4 paper is just ~three months old
Many of these techniques haven't been published very long ago - it often takes a good 6-8 months for techniques to percolate. But also, they come at a complexity cost and, seemingly, also at a stability cost.
Also potentially a performance (in terms of output quality) cost. DeepSeek is cheap on a per token basis but lags behind in the benchmarks, perhaps it was a calculated tradeoff.