Comment by steeleduncan
1 day ago
12GB is not a lot, you are stuck on low quantisations of older models with that. What exactly did you try?
The newer/larger models are good though, not as good as Claude, but not every task requires a model of that level. Maybe yours do, but for simple coding tasks, summarising text, etc a small local model runs just fine, and is a lot cheaper than Claude
What graphics cards have more than 12GB, and how much does it cost to buy one? What about for others saying you ideally want 32GB? (That split will determine if a 5080 will do it)
32GB would probably get you qwen3.8-27b at a reasonable quantisation, which benchmarks similar to opus 4.6
You can try that model (and many others) using a cloud host first, and then only if you like it do you need to buy something to run it.
> What graphics cards have more than 12GB, and how much does it cost to buy one?
RTX 5070 Ti and RTX 5080 both have 16 GB.
RTX 5090 has 32 GB.
Good luck finding any of them for reasonable prices, though. I paid $3,000 for my 5090 early last year. I winced when I bought it then. Today, it's like $7,000, and I wouldn't even consider it at that price.
But I'm mostly a gamer that also does local AI as a hobby. If the RTX 5090 was $7,000, I'd still be on my RTX 3080 and still just not started doing local AI and have to turn down the graphics settings in Cyberpunk 2077.
The RTX 5090 is the ideal, but it's not the only 32GB GPU that can offer workable performance; the AMD R9700 is catching up.
You can surely build an LLM server with an R9700 and 32GB of mainboard RAM for less than the 64GB M5 Max Apple Studio, which is probably the next most competitive model up. It'll be better than a Strix Halo but less flexible.
5 replies →
there is also a version of the 5060ti woth 16GB. i got lucky a few months ago and got it for ~380€, but i think you can still get it for around 800, definitely under 1k