Comment by fennecbutt

4 hours ago

I mean I haven't checked but I'd guess it'd be possible to save a kv cache to an ssd or ram and then pull it back into vram when needed.

0 comments