Comment by the__alchemist
1 day ago
That is a shift of comparisons though, right? This thread was about running locally, explicitly as an alternative to cloud services. I would guess few people have graphics cards with > 12GB ram.
1 day ago
That is a shift of comparisons though, right? This thread was about running locally, explicitly as an alternative to cloud services. I would guess few people have graphics cards with > 12GB ram.
A few more will have Macs with 48GB RAM though. (Indeed if you push it a little, 36GB unified is possibly enough for Muse Glimmer; I have not tested this theory)
There is, for sure, a limiting factor in the x86 architecture, but I think the RTX Spark will change that a bit, and maybe AMD's subsequent Ryzen AI/Strix Halo chips.
AMD Ryzen AI/Strix Halo is for running locally on your hardware. Openrouter is for evaluating the models you'll be running locally, before buying the hardware.
In any case, 128-256GB unified memory systems running MOE models is the "cheap" option, RTX6000 is the step above it.
> 128-256GB unified memory systems running MOE models is the "cheap"
I'm not familiar with these. How much does one cost, and from what store / what product name?