Comment by armcat

This is fantastic. I feel the KV cache system is basically a must for any serious inference hosting, both for the provider as well as the consumer. I guess they have this but just haven’t open sourced it yet.