Comment by russli1993

could push kv-cache to off chip cheaper storage like dram. Inference engine manages decode for a single user to run to completion before decode a different users so kv-cache fetch from dram to sram is limited.