Comment by gitpusher42
5 hours ago
Size reduction is mostly based on Experts size. And it is limited by SSD speed. Check for Colibri and Flash-Moe, they are doing similar things with bigger models, but tok/s is not high
5 hours ago
Size reduction is mostly based on Experts size. And it is limited by SSD speed. Check for Colibri and Flash-Moe, they are doing similar things with bigger models, but tok/s is not high
No comments yet
Contribute on Hacker News ↗