Comment by wernam
1 day ago
12GB GPU is useless. The good experience is with 128-256GB unified memory architectures like strix halo etc. You don't need to buy any hardware, you can experience the latest open weight MOE (flash) models from DeepSeek, qwen, or GLM on openrouter using a local agent like kilo.
A 3060 + good chunk of system RAM can run qwen3.8 flash next.
That is a shift of comparisons though, right? This thread was about running locally, explicitly as an alternative to cloud services. I would guess few people have graphics cards with > 12GB ram.
A few more will have Macs with 48GB RAM though. (Indeed if you push it a little, 36GB unified is possibly enough for Muse Glimmer; I have not tested this theory)
There is, for sure, a limiting factor in the x86 architecture, but I think the RTX Spark will change that a bit, and maybe AMD's subsequent Ryzen AI/Strix Halo chips.
AMD Ryzen AI/Strix Halo is for running locally on your hardware. Openrouter is for evaluating the models you'll be running locally, before buying the hardware.
In any case, 128-256GB unified memory systems running MOE models is the "cheap" option, RTX6000 is the step above it.
> 128-256GB unified memory systems running MOE models is the "cheap"
I'm not familiar with these. How much does one cost, and from what store / what product name?
Would you talk to me a mo about Kilo compared to Pi?
I am digging in on Pi (in lovely Paseo and exploring Pendant when in VS Code).
What would I find better or worse in Kilo?