Comment by dofm
18 hours ago
The RTX 5090 is the ideal, but it's not the only 32GB GPU that can offer workable performance; the AMD R9700 is catching up.
You can surely build an LLM server with an R9700 and 32GB of mainboard RAM for less than the 64GB M5 Max Apple Studio, which is probably the next most competitive model up. It'll be better than a Strix Halo but less flexible.
Of interest:
Compared to $20 or $100 per month. My thought process reading this thread:
Baseline of $20 or $100/month Claude or OpenAI subscriptions. Alternative posed of running models locally. A guy says he did it and results are lackluster on a 12GB video card. A number of people post that that card is insufficient; use 32GB+ cards. I look up prices of 32GB cards, and they are all very expensive. So from a practical perspective, people are suggesting you spend thousands of dollars on a video card to get results that are almost as good as the subscription plan they are on. It doesn't add up, unless one of my assumptions is incorrect.
The assumption that is probably incorrect is that the current prices will last. A 4090 should not cost that much, and most of your Claude usage is paid by an investor. Neither of these things will be true in a few years.
Having a 4090 at home running agents all day is not a $20pcm level of usage, so take the $100 plan. Looking at the S-1 filing, today they are not even close to breakeven. It is reasonable to assume that will become a $300-400 pcm plan.
The launch MSRP of a 4090 was $1600. There has been significant inflation since then, so say $2000. Eventually supply will meet demand and you'll be able to buy one for that price again, or lower due to the economies of scale they must have now.
So, in 2-3 years time once it has all settled, you pay off your 4090 by saving 5-6 months of Claude usage. Maybe the local model isn't good enough for complete replacement, but all it needs to do is drop you to the $20 plan to be worth it. I think this is a large part of why investors are getting jittery now.
Right. Nobody says it's necessarily economically the best option. But it's also an asset that has alternative applications and barely depreciates!
Right now you could extensively use an M5 Max for a year and sell it for almost exactly the same amount at the end. Maybe even more, at the current rate. And it can do other things for you: host your VMs, edit high def video, whatever. Even if it stops being useful to serve LLMs, it might be useful to run as a server for other tasks for a whole decade. It will be a fast, supported Mac for seven years. For the R9700, maybe gaming?
Depends what you want, doesn't it? Or are we all expected to conform?
For me personally, right now I have a secondhand M1 Max macbook pro, and that is my local AI machine, because my needs are exploratory and I want to run it here. I want to learn by comparing models. I want to learn how to run them and advise others how to (local inference is not a zero-sized market).
My next step up will be to take the apps I'm looking at building that run against these small models, and host those apps online with some sort of cloud inference — openrouter, digitalocean, cloudflare — for those same models.
If I ever find I want to vaguely describe some desktop application before bed and come down the next morning to an aggravating slop version of it written in Rust, I'll consider Claude.
For a local non-laptop, I've looked at the M5 Max and the R9700. I've also looked at the M5 Pro, which in practice is about as fast at token generation as my M1 Max (because the M1 Max's memory bandwidth is strictly theoretical), and smokes it at prefill. That alone might be enough for a "leave it running" mini AI box, and it will be a nice desktop machine. I just don't need it enough yet.
As an aside there's this thing — somewhat interesting, though constrained:
https://www.olares.com/one
Uses the 5090 mobile chip, which is pretty good speedwise (about the same as the R9700) but 24GB. Small, tidy, on-prem. I think they have missed the boat on pricing, but if they can ship in volume, maybe not.
Great insights! And I'm kicking myself for not buying a 4090 3 years ago for $1600, shipped direct from nvidia... it seemed like too much to pay for a GPU at the time!
1 reply →