Comment by lostmsu
15 hours ago
They don't have cache (e.g. KV cache). But they write down what you sent earlier to say they cached it! To still bill the same as uncached later (because they didn't actually cache it)!
15 hours ago
They don't have cache (e.g. KV cache). But they write down what you sent earlier to say they cached it! To still bill the same as uncached later (because they didn't actually cache it)!
More precisely they can't cache it.
Yes. Their architecture recomputes every time so at 150k context every request will have to spend 1.5 min waiting for the model to reread the context.
Say avg model response length is 1024 tok. At 50 tok/s normal providers do your turn will only take 20s (vs Cerebras 101s) and will cost 20x less. That time and cost is per single tool call.