But its also a 800B sized model running on a ram constrained system with no GPU.
Techniques, GPUs, more ram, and faster disks can always speed it up. But the point being is they run on low end machines now. Its now an optimization problem, not a possibility assessment.
Already is possible. On a machine with 32GB ram, and NO gpu. Just need a large SSD or NVME. Streams from disk to memory.
https://github.com/JustVugg/colibri
Painfully slow tok/s though.
I didn't say it was fast!
But its also a 800B sized model running on a ram constrained system with no GPU.
Techniques, GPUs, more ram, and faster disks can always speed it up. But the point being is they run on low end machines now. Its now an optimization problem, not a possibility assessment.
Isn't it more s/tok?
That's fucking awesome!