Comment by dave7
20 hours ago
12GB GPU is insufficient for most purposes yes. Even 16GB is a notable improvement, but yes 32GB cards such as the 5090 or R9700 are a world of difference.
20 hours ago
12GB GPU is insufficient for most purposes yes. Even 16GB is a notable improvement, but yes 32GB cards such as the 5090 or R9700 are a world of difference.
Given we're discussing a cost comparison (i.e. cutting a Claude subscription in lieu of running locally), how many months of claude $100 or $20/month plans would it cost to buy a 5090? I am showing it as 88 and 440 respectively. I.e. a 7 and 36 year payoff. Not even close to being worth it.
You also get to keep your data private (so you should compare it to api pricing), there are no forbidden topics (i just did a pentest on my webapp), and multiple people can use it. Basalt does 150 tok/s on a 5080
Currently, it's not worth it because:
"SemiAnalysis, a research and analysis company specializing in the semiconductor and AI industries, found that a $200/month subscription plan is far more generous than previously thought. According to its calculations, while it's widely believed that a $200/month plan maxes out at roughly $2,000/month in tokens (assuming API pricing), the actual numbers are closer to $14,000 for ChatGPT and $8,000 for Claude. "
https://cybernews.com/ai-news/chatgpt-claude-usage-costs-ope...
This will change in the future.
The Claude plans are probably still heavily subsidized. Expect prices to increase, especially after the IPO. Self-hosting models for personal use is probably overkill, but I can see companies doing it pretty soon since they can probably run 24/7 workloads and fully utilize the hardware.
Great point. And, 3 years ago, a 4090 cost $1600, so maybe that will happen again some day. (Same/similar card, or a future card. I remember when GPUs used to get better every few years instead of staying roughly the same but costing several times the original amount!)