Comment by labcomputer

4 hours ago

SSD token caching (like oMLX does) goes a long way for prefill. Typical miss rates are under 5% in long coding sessions, which is mostly the output of a turn's tool use.

0 comments