Comment by FuckButtons
1 day ago
Not sure why ds cache costs are confusing? Their kv cache is absurdly compressed compared to almost everyone else.
1 day ago
Not sure why ds cache costs are confusing? Their kv cache is absurdly compressed compared to almost everyone else.
It is confusing because they are the only ones doing so whereas everyone has an incentive to apply similar techniques.
DS4's architecture was redesigned for cache efficiency. It's the model itself that is just way more cache efficient, in addition to a distributed KV cache (which they open sourced last year). You could run your own LLM provider just for DS4 models, recreate their setup, and get the same cache efficiency, but it would only work for that one model family.
Other models would also need to be rearchitected to get the same benefit, and I suspect they don't want to, because it seems to create both larger models and slightly less intelligence (DS4* has smaller active experts to get faster results).