Comment by whimsicalism

would loading the KV cache from disk be faster than just recomputing it?

imo the discontinuous segments bit would not work because of the causal dependence in transformers + RoPE as you mention, but maybe could be possible