Comment by swiftcoder
2 days ago
> it would likely take years to spend $4000 (plus the real cost of electricity)
Since that cluster only yields 20-30 tok/s on that size of model, at least a decade before the hardware breaks-even with current token costs, and that's not counting electricity. Assuming continued downward pressure on token prices, and the cost of electricity, it never pays for itself.
I don't understand how people don't consider this.
Plus you're spec'd out of near-SOTA level in months.
The only reasons to actually do this are a) you have a lot of dispensable income and are a hobbyist/tinkerer, b) you have real, legitimate privacy concerns or, relatedly, c) you're doing something you don't want to get flagged
Not everything is about pure cost. Maybe I don't want to sell my soul supporting the frontier labs because they are straight up pure evil?
I barely see a difference between buying the hardware that feeds (and often colludes with) those labs, at least not as a moral stance.
Even if you trained your own model, you'd be committing some of the same sins, paying for the same hardware that drove it, etc. But if you're using some open model, you're standing on the shoulders of the same corrupt giants.
I feel like when people say this is due to moral reasons, it's to justify an expensive hobby.
1 reply →
"you don't want to get flagged"
Ding!
What is actually getting you flagged by the openweights inference providers? Thus far I haven't hit any of the reverse engineering or infosec guardrails that Anthropic is so keen on
2 replies →
As a counterpoint, my homelab/home-LLM hardware has appreciated in value by about 60% since I bought it.
Of course, it's not real unless I sell, and the value will eventually go down, but so far I have significant paper profits.
Also, DeepSeek token prices are continuing to _increase_, not decrease.
> DeepSeek token prices are continuing to _increase_
One increase does not a trend make. And the current crop of models are now undercutting deepseek flash...
You can't possibly think that it's going to get cheaper and cheaper to pay for tokens though. Right? Have you seen what's happening with Codex/Claude subscriptions? Deepseek raising API prices.. We've been getting subsidized tokens for some time now and as the hardware costs skyrocket these labs/people with inference compute are going to continue to clamp down.
5 replies →
> it never pays for itself.
Exactly; its a development box for fiddling with GPU hardware with a large amount of video-addressable memory. It's not an inference box, really, though it's neat that I can at all!