Comment by labcomputer
3 hours ago
SSD token caching (like oMLX does) goes a long way for prefill. Typical miss rates are under 5% in long coding sessions, which is mostly the output of a turn's tool use.
3 hours ago
SSD token caching (like oMLX does) goes a long way for prefill. Typical miss rates are under 5% in long coding sessions, which is mostly the output of a turn's tool use.
No comments yet
Contribute on Hacker News ↗