Comment by gaigalas
2 days ago
Waiting for the MTP version to pop up on Unsloth. Speculative decoding makes a huge difference.
Been running quantized 3.6 at 110t/s on a cheap 5060Ti and quite happy with it. If 3.8 improves on it, it would be awesome.
Good news, MTP support is already included in this release. Not sure why they haven't made this clearer.
I don't see an MTP entry on Unsloth though. Maybe it's not available in a lower quant I need for my poor GPU.
It's included in the model gguf itself.
1 reply →