Comment by the__alchemist
21 hours ago
What graphics cards have more than 12GB, and how much does it cost to buy one? What about for others saying you ideally want 32GB? (That split will determine if a 5080 will do it)
21 hours ago
What graphics cards have more than 12GB, and how much does it cost to buy one? What about for others saying you ideally want 32GB? (That split will determine if a 5080 will do it)
32GB would probably get you qwen3.8-27b at a reasonable quantisation, which benchmarks similar to opus 4.6
You can try that model (and many others) using a cloud host first, and then only if you like it do you need to buy something to run it.
> What graphics cards have more than 12GB, and how much does it cost to buy one?
RTX 5070 Ti and RTX 5080 both have 16 GB.
RTX 5090 has 32 GB.
Good luck finding any of them for reasonable prices, though. I paid $3,000 for my 5090 early last year. I winced when I bought it then. Today, it's like $7,000, and I wouldn't even consider it at that price.
But I'm mostly a gamer that also does local AI as a hobby. If the RTX 5090 was $7,000, I'd still be on my RTX 3080 and still just not started doing local AI and have to turn down the graphics settings in Cyberpunk 2077.
The RTX 5090 is the ideal, but it's not the only 32GB GPU that can offer workable performance; the AMD R9700 is catching up.
You can surely build an LLM server with an R9700 and 32GB of mainboard RAM for less than the 64GB M5 Max Apple Studio, which is probably the next most competitive model up. It'll be better than a Strix Halo but less flexible.
Of interest:
Compared to $20 or $100 per month. My thought process reading this thread:
Baseline of $20 or $100/month Claude or OpenAI subscriptions. Alternative posed of running models locally. A guy says he did it and results are lackluster on a 12GB video card. A number of people post that that card is insufficient; use 32GB+ cards. I look up prices of 32GB cards, and they are all very expensive. So from a practical perspective, people are suggesting you spend thousands of dollars on a video card to get results that are almost as good as the subscription plan they are on. It doesn't add up, unless one of my assumptions is incorrect.
4 replies →
there is also a version of the 5060ti woth 16GB. i got lucky a few months ago and got it for ~380€, but i think you can still get it for around 800, definitely under 1k