← Back to context

Comment by tasuki

6 days ago

Ok, say I have 14GB VRAM. What is the tradeoff between using 9B with 8-bit params vs 27B with 3-bit params?

3 comments

tasuki

Reply
causal  6 days ago

You COULD even do Qwen3.5-35B-A3B-GGUF.

UD-IQ3-XXS is only 13.1GB, which might outperform both in both intelligence and certainly speed (only 3B activated): https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF

To accommodate cache you will need to offload a few feed-forward layers to the CPU. Will still be quite fast.

Edit: Actually 27B does a little better than 35B on most benchmarks- 35B will still be much faster.

causal  6 days ago

3-bit 27B will almost certainly be better. 4-bits is usually the limit below-which you start to see more steep drop-offs, but you also get diminishing returns above 6-bits. So I'd still rather pack in more params at 3-bits.

9B will be faster, however.