Comment by jedbrooke
3 days ago
I hope the bonsai team makes another 1bit quant of this model (or releases code/instructions on how to do it), using the Qwen3.6 27B on my 16GB mac mini has been wild . The 1bit quant feels like opus level… for the first couple turns. Then it has trouble eg switching from plan mode to act mode. This is mostly mitigated by starting a new session. (tbf this limitation is called out on the hf page)
I saw unsloth has 1bit quants too so I might check that out, anybody have experience with those?
16GB mac mini - what chip? m4 pro i assume?
nope just my normal m2 mac mini. I bought it just as a normal computer to do my taxes and whatever, so it’s mind blowing that I can run this kind of AI workload on it. Well, “run” might be generous, it gets like 3-5tok/s
I’m working on a setup that’s more geared towards running tasks overnight so the slow tok/s doesn’t matter as much
ahh gotcha. that speed sounds about right. i am running qwen 3.6 35bA3B(group-32 int4) on my m3 air (16gb) and get about 8 tokens per sec. i have not yet tried the 3.6 27B.
Sounds like you need to check what the max context is set to ...
100k is all the context I have ram for, this is with any auto-compact turned off. This is using Cline in vs code. I’m sure I could tune the system prompt and mode switching more to work better with this specific model, but I haven’t gone down the custom harness rabbit hole yet.
And this is also specifically for the 1bit quant version. I don’t think the fp8 or even fp4 versions have this issue, but I haven’t tried those much