Comment by gitpusher42
4 hours ago
Size reduction is mostly based on Experts size. And it is limited by SSD speed. Check for Colibri and Flash-Moe, they are doing similar things with bigger models, but tok/s is not high
4 hours ago
Size reduction is mostly based on Experts size. And it is limited by SSD speed. Check for Colibri and Flash-Moe, they are doing similar things with bigger models, but tok/s is not high
No comments yet
Contribute on Hacker News ↗