Comment by snthpy
6 days ago
What is the best way to deploy these on CPUs, arm64 ones in particular?
I'm interested in the CPU inference application of these models with things like the FairyFuse kernels.
I've tried trillim previously but was disappointed that i got higher tok/s just with similar sized models through ollama using just Q4_K_M quants.
I see there is bitnet.cpp and litespark-inference. What else should i look at?
Llama.cpp should be rolling out support for this soon if they haven't already. Cactus is a but more targeted for efficient ARM execution, but I haven't been keeping up with what all they support. Would be worth a feature request if they aren't working on it yet.