Comment by Saris

I run it on a 12GB 4070 with 32GB system RAM. 35B A4B means only part of the model is active at a time so it takes a lot less VRAM than a dense 35B model would.

The main thing in LM studio (or whatever software you use, assuming it has fairly up to date stuff and exposes the toggles) is to offload MoE layers to the CPU, and use K/V cache quantization at Q8_0 or Q4_0.

Since you have more VRAM than I do, you could probably get away with MoE offload of like 15-20 so some remains on the GPU.

Just make sure GPU offload is turned all the way up. And I use 64k context size, although with 16GB VRAM you can probably do more.

You can find the best performance spot by playing with MoE offload until you find the number that gives the highest tok/s on your hardware.