Comment by kristianp
4 hours ago
They've shown the calculation for number of concurrent requests limited by kv cache size. I guess for each new request loaded into the queue, the kv cache for that context will need to be loaded from a supporting storage server, that will need to have a lot of ram to store the active caches of multiple requests?
Less "storage server" and more "attention server".
A server that both holds KV cache in its own memory, and uses its own compute to perform attention operations. That way, you don't have to transfer cache entries server to server for every attention sweep - only the "requests" and "responses". Attention bandwidth stays server-local.