Comment by data-ottawa

14 hours ago

We should still expect significant performance improvements.

I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.

10 comments

data-ottawa

Reply
hedgehog  14 hours ago

The active param count is so small I'm not sure how much advantage MTP will have, the current llama.cpp does load the ngram embeddings but I haven't verified it uses them. I expect to redeploy all this stuff every few days as the tooling gets improved.

  • cmrdporcupine  13 hours ago

    My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload.

    It's not world changing, but at those speeds I'll take anything I can get.

    (The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).

    https://github.com/rdaum/eider/

    • hedgehog  13 hours ago

      It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.

      7 replies →