Comment by Gareth321

3 hours ago

I would love to hear more about your setup. Which hardware and models? How much context can it handle? How does it compare to frontier models?

2 comments

Gareth321

Reply
_ache_  1 hour ago

I mostly run Qwen3.8 Flash Next on 2xR9700 (but Intel Arc Pro B70 may be better currently, or 4x3090 is even better but out of budget, 32Gio Nvidia GCs on ebay are also usable). I have 128Gio DDR4 (3200Hz), but 64Gio is enough, even 32Gio maybe, I just reuse it from an old project.

You may need a AMD EPYC processor and associated motherboard, don't forget a good PSU at least 1200W (300W processor + 400W by card to support spikes of consumption). On some setup (~15k€) you can use 4 CG to go up to 128G of VRAM.

I'm not using a EPYC processor for now, so I get 170tps on decode but very slow prefill, the bottleneck is the PCIe exchange between the cards, I need to setup P2P (it means what you think it means, for GCs).

My setup supports full context (2*18 bytes so 256K), I didn't setup YaRN to get 1M of context (2*20 bytes), but it's an option with 128Gio of RAM, the visual component is extracted from RAM so a little slower. I'm not using multiple agents but some people does, it's god like with 4 graphics cards, on 2 only, it's relatively quick but limited to single user and up to 3 to 4 concurrents requests only.

It's not frontier, but look AA of QFN, it's good enough. DeepSeek V4.1 can run on my setup, but it's slow. On dual B70, there is some PoCs to make it usable (>60tps). It improves every days.

If you need more information, look for local setup users on the Web. There is a lot of people sharing information about benchmarks and setups. L80 is my reference for now.