Comment by ch_sm
7 days ago
Nice, I tried it too with oMLX — agreed, it seems very capable for coding! Was slightly underwhelmed by performance though. I got about ~24 t/s on the ternary version on my M2 Max 64. That’s quite a bit slower than Qwen A3B 35B (4 bit unsloth). How was perf for you?
i'm getting similar toks on my 36gb m3 max. new omlx release (0.5.2) has dedicated decode kernels for the bonsai models — see if this improves perf for you.
I had roughly the same performance on M3 Max 64: ~30 tokens/sec. Which is not terrible, but with the latest Lightning MTP optimization I am getting ~100 tokens/sec with Qwen 3.6 35B-A3B.
WHAT. that‘s amazing. thanks so much for sharing, i‘ve been looking for ways to speed a3b up for days. It‘s 11pm here but i‘ll try this right now
I’m using the new oQe quants in case that matters.
2 replies →