Comment by labcomputer
5 hours ago
SSD token caching (like oMLX does) goes a long way for prefill. Typical miss rates are under 5% in long coding sessions, which is mostly the output of a turn's tool use.
5 hours ago
SSD token caching (like oMLX does) goes a long way for prefill. Typical miss rates are under 5% in long coding sessions, which is mostly the output of a turn's tool use.
No comments yet
Contribute on Hacker News ↗