Comment by Archit3ch

2 days ago

You can keep the KV cache in (possibly Unified) RAM to avoid SSD writes entirely. Not sure if it would fit on a 32GB laptop, though.

0 comments