← Back to context Comment by saagarjha 8 hours ago Sure, but any classical attention mechanism is quadratic in context length. 1 comment saagarjha Reply zozbot234 7 hours ago But text generation is quadratic after the KV cache optimization. If every decode step now has to recompute KV cache including its latest and most expensive tokens (even with a quick, "draft" model) that's even worse.
zozbot234 7 hours ago But text generation is quadratic after the KV cache optimization. If every decode step now has to recompute KV cache including its latest and most expensive tokens (even with a quick, "draft" model) that's even worse.
But text generation is quadratic after the KV cache optimization. If every decode step now has to recompute KV cache including its latest and most expensive tokens (even with a quick, "draft" model) that's even worse.