← Back to context Comment by tommica 20 hours ago Oh, that is a useful rule to know! Thanks! 1 comment tommica Reply cyanydeez 12 hours ago That's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go.And the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache.Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.
cyanydeez 12 hours ago That's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go.And the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache.Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.
That's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go.
And the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache.
Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.