Comment by hedgehog
10 hours ago
The active param count is so small I'm not sure how much advantage MTP will have, the current llama.cpp does load the ngram embeddings but I haven't verified it uses them. I expect to redeploy all this stuff every few days as the tooling gets improved.
My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload.
It's not world changing, but at those speeds I'll take anything I can get.
(The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).
https://github.com/rdaum/eider/
It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.
Probably. I've spent zero time with optimization at this point. Code is all new this morning.
Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk.
Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?
6 replies →