Comment by cmrdporcupine

13 hours ago

My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload.

It's not world changing, but at those speeds I'll take anything I can get.

(The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).

https://github.com/rdaum/eider/

8 comments

cmrdporcupine

Reply
hedgehog  12 hours ago

It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.

  • cmrdporcupine  12 hours ago

    Probably. I've spent zero time with optimization at this point. Code is all new this morning.

    Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk.

    Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?