Comment by the_duke
6 days ago
What context length and related performance are you getting out if this setup?
At least 100k context without huge degradation is important for coding tasks. Most "I'm running this locally" reports only cover testing with very small context.
Long context degradation is a problem with the Qwen3.5 models for me. They have some clever tricks to accelerate attention that favor more recent context.
The models can be frustrating to use if you expect long contexts to behave like they do on SOTA models. In my trials I could give them strict instructions to NOT do something and they would follow it for a short time before ignoring my prompt and doing the things I told it not to do.
Q4 quants on 32G VRAM gives you 131K context for 35BA3B and 27B models who are pretty capable. On 5090 one gets 175 tg and ~7K pp with 35BA3B, 27B isaround 90 tg. So speed is awesome. Even Strix 395 gives 40 tk/s and 256K context. Pretty amazing, there is a reason people are excited about qwen 3.5