Comment by memre12
5 hours ago
Impressive if the numbers hold. Would love a table with per-token bytes read, measured SSD bandwidth, and cache hit rate—those four numbers would make the tok/s claims land better.
5 hours ago
Impressive if the numbers hold. Would love a table with per-token bytes read, measured SSD bandwidth, and cache hit rate—those four numbers would make the tok/s claims land better.
I double checked m2 logs. Cache hit rate is about 59-69%. 250-320MB went through `pread` per generated token. It is 3gb/s during this i/o phase.