Comment by Grombobulous
20 hours ago
If you have enough system RAM (64GB), you can run Qwen 3.8 Flash Next with a GPU as small as 12GB, this repo is pretty amazing:
https://github.com/Niko1221/Strata/
My setup is a Ryzen 3600, 90GB DDR4 allocated to a VM (don’t even really need that much, just had it available, special thanks to the year 2024), and an RX 9060XT 16GB ($500 GPU).
I get around 40-60 tok/s depending on the quantization level I choose.
As you can see, the CPU is nothing special, don’t even need DDR5.
Yes it runs on all levels of hardware - with the latest Strata my 5090 and 64GB DDR5 gets 200-240 tok/sec decode, 6500 tok/sec prefill. Around $0.09/M tokens in power usage (have excess solar capacity).
Jealous, lol. But I did spend a grand total of $500 for my setup. The rest of the server was built purchased with the use case of running game servers.