Comment by otterley
8 hours ago
How much did you shell out for this setup? What’s the TTFT and the output tokens/sec rate? How much electricity does it consume?
8 hours ago
How much did you shell out for this setup? What’s the TTFT and the output tokens/sec rate? How much electricity does it consume?
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
Would you use hosted models if they promised anonymity and no log retention and servers were physically located inside jurisdictions that respect privacy (Switzerland, Norway or similar, I'm not into law) if that meant better price/performance ratio than your current rig?
I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
24 replies →
If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt.
9 replies →
LLM subscriptions are heavily subsidized. I’d be surprised if Claude 20x costed the same after a decade.
6 replies →
But what if you become unable to pay for it ? What if:
- they raise the prices 10x
- go bankrupt
- ban you or your country
- decide not to help you with what you want to do for opaque reasons
- you are in jail because the model halucinated something and the company promptly reported you to the police
Better avoid all off these and go local IMHO.
I was thinking the economics of it the same way.. if the AI companies are burning debt and VC money to give me cheaper tokens why not take them for it? I can always rent a GPU server for those personal, private AI needs.. heck even run a decision model to route them to the rented GPU servers either on demand or scheduled for few hours in a day…
It doesn’t have to be black or white…
For my toy projects, adventures, learning exercises, and throwaway MVPs I will prefer the subscription plans… they are cheap for what they offer today.
4 replies →
Keep in mind that the prices of the big AI companies are currently mostly loss makers. Their plan is to get sufficient market saturation and dependency of their clients and if they achieve that, they will hike up the prices quite a bit.
(Of course this all depends on if their plans work out, but if not then the smaller local models will probably play a big role in the failure of their plan)
4 replies →
Do you take into account the token allowance? For a heavy user this would be the equivalent of many subscriptions and they add up.
If you compare to API prices the value difference is not as big.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It's hard to go with anyone else unless you're rich.
If you compare to API prices the value difference is not as big.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It'd stick with it too
The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
1 reply →
If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse.
3 replies →
The benefit of the agent working firstly for you rather than for Anthropic or OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users.
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance.
> It is not cheaper than using a frontier lab
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
True, they're subsidized, I'm not. But I don't have their utilization so I figure it would always be more expensive to run this locally.
Did you get insurance on all this stuff? I imagine home insurance would not qualify for a bespoke home-datacenter.
Not OP. But I’ve done some calculation here https://localaicost.com/
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
V100S is a very good card for some workloads. And they were cheap as well.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
1 reply →
[dead]