Comment by glassresearch
7 hours ago
I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
7 hours ago
I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
Also no political censorship, and it doesn’t report you to the police when it suspects you did something illegal. That really happened, by the way.
Now I want a local LLM server :( $25k is just so much money.
No political censorhip? Try asking Chinese models about Taiwan.
Some censorhip and bias towards certain ideology is baked into those models.
While it is not a problem for programming tasks, I would never use it for certain other tasks.
1 reply →
If you already have a machine, throwing an r9700 in it is one of the cheapest ways to get qwen3.8 27B Q6 with 100k+ context working.
Sadly, the r9700 has gone up in price, but it's still the cheapest route. I use It on an old x99 Linux desktop and it achieves 50 tps decode and 1000+ tps prefill.
1 reply →
I'm building a local LLM server myself. For about 8k€ you can have 170tps, ~6000 prefill.
A lot more with Haiku equivalents (Qwen3.8 27B) or gemini. It's not frontier, but not that far away.
1 reply →
>and it doesn’t report you to the police when it suspects you did something illegal
This is an alignment issue. OpenClaw with a local LLM with an email you setup for it could contact the police.
1 reply →
MacBooks with large RAM and optimized for local models should be Apple's medium term goal.
1 reply →
> My local model never says no.
> I don't get to hear that it's not a doctor and I should call 911.
https://en.wikipedia.org/wiki/Darwin_Awards
It's pretty easy for a reasonable person to determine when to call 911 and when to ask an LLM. Typically when I ask an LLM, it is for something I am not willing to escalate further at the moment.
Anthropic has a significant interest in reducing liability, which in this case means mostly blocking this use case.
Exactly, that's the spirit. I pool the resources with some friends as well.
> My local model never says no. I never get banned.
So much this. Plus everything else you mentioned. The freedom and control is real.
isn't it the same using the same model you host locally but through a cloud provider? does the $ math favour a local setup then?
>I don't get to hear that it's not a doctor and I should call 911.
Local models can do this and with tool use they can even call emergency services for you on your behalf.
I think both of y’all glossed over the $25k of it all pretty quickly. This isn’t remotely an option for anyone.
AnyONE. But if you got some friends, and they got friends, then that $25k becomes smaller and stomachable. My favorite rumor out of LA is groups of friends with a Lamborghini Urus that they timeshare. No idea if it's true, but if car people can get together to buy/lease a car that's MSRP is $345k, I think $25k should be an option for anyone with friends. (Hypothetically. I've read a lot about this friend thing on the Internet!)
2 replies →
Its a drop in the bucket for many businesses though.
1 reply →
If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt.
That NDA case is the real argument, more than price. Token prices keep dropping anyway, but "this data can never leave the building" doesn't change with a price cut ;)
Okay man I've got to ask, what kind of data are we talking about? Legal? Medical?
I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.
http://www.modularcompany.com/
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
5 replies →
This trope minimizes so many downsides of "frontier" models. When you can't do what you need to with a model it really isn't valuable - which is why subscriptions still exist. As soon as you're forced to pay actual token costs people who keep saying this will likely wish they had invested in something else earlier.
It's also very clear that providers are running variants of models, especially at the SotA levels. I wouldn't be surprised to find out at some point that 1-2 months after release OAI and Anthropic just run "good enough" quants to save themselves on inference (since they're losing money hand over fist). Another upside of local models is that I don't need to worry about model degradation at any given point.
LLM subscriptions are heavily subsidized. I’d be surprised if Claude 20x costed the same after a decade.
Inference prices are going only down. It could be that some top subscription costs more in a decade, but I fully believe a full array of competent models will be extremely cheap.
The inference is profitable. It’s the training which is really expensive and not being reflected in the subscription prices.
Doesn't deepseek make $38 per GPU-hour? As in they can actually serve profitably?
Deepseek doesn't have subscriptions, you can only pay per token / API prices.
2 replies →
But what if you become unable to pay for it ? What if:
- they raise the prices 10x
- go bankrupt
- ban you or your country
- decide not to help you with what you want to do for opaque reasons
- you are in jail because the model halucinated something and the company promptly reported you to the police
Better avoid all off these and go local IMHO.
I was thinking the economics of it the same way.. if the AI companies are burning debt and VC money to give me cheaper tokens why not take them for it? I can always rent a GPU server for those personal, private AI needs.. heck even run a decision model to route them to the rented GPU servers either on demand or scheduled for few hours in a day…
It doesn’t have to be black or white…
For my toy projects, adventures, learning exercises, and throwaway MVPs I will prefer the subscription plans… they are cheap for what they offer today.
From this article alone it should be obvious that the long-repeated myth about frontier inference being subsidized is just that: a myth.
It's training and tuning new models that cost them money. And that's why they are trying to scare people into thinking AI will kill us all in the next 5 years; they want bans on the open-weight models because they know they won't be able to compete with them.
A few days ago we got the news that DeepSeek makes 38 dollars profit per card with their off peak pricing (in the inference server release discussion).
If that is true, either the American labs are not optimizing correctly or they make a ton of money with inference.
1 reply →
Yeah, the way I see it short term is take the free/subsidized tokens while we can and the further out we go the cheaper compute will get for local setups.
Keep in mind that the prices of the big AI companies are currently mostly loss makers. Their plan is to get sufficient market saturation and dependency of their clients and if they achieve that, they will hike up the prices quite a bit.
(Of course this all depends on if their plans work out, but if not then the smaller local models will probably play a big role in the failure of their plan)
People keep repeating this, but it is only remotely close to true for the literal free plans and the small subset of consumers maxing out the pro plans.
At API pricing (i.e. what all enterprise customers are paying), inference is profitable, even if they're paying usurious rates on cost of capital for capex. Where the frontier labs are getting screwed is that they're in a Red Queen's race with each other, spending tens of billions of dollars per year training the next frontier model, and not being able to recoup the investment on that model before it's obsolete.
That's why you see the CEOs calling for government intervention to "pace the frontier": it lets them reduce investment on training without risking that their competitors keep going and make them obsolete. Unfortunately for them, the Chinese labs exist, and would love nothing more than to take the frontier, so they're in a bit of a pickle.
The people I personally know who signed the "pacing the frontier" petition (including me), or who called for it in other ways, are sincere in their concerns that things are moving too fast for safety.
1 reply →
> inference is profitable
People keep repeating this, but it is only remotely close to true if you evaluate inference in a vacuum without any of the other supporting costs to facilitate that capability.
Just a few years ago people were saying the same thing about WeWork: "leasing is profitable, the unit economics of individual locations is net positive", and yet...
Do you take into account the token allowance? For a heavy user this would be the equivalent of many subscriptions and they add up.
If you compare to API prices the value difference is not as big.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It's hard to go with anyone else unless you're rich.
If you compare to API prices the value difference is not as big.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It'd stick with it too
The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
> The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
But, there is no need to make that bet - when prices increase, I'd look into renting a previous-gen GPU on a cloud provider. If the online GPU providers raise prices, then I would look into purchasing for on-site.
Until that happens, I'm not making any bets, because if all the token providers go bust GPU rentals are going to fall in price, and if they don't go bust after raising prices, I can still get years of GPUs for self-hosting at the price of a self-owned rig.
If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse.
Would you really expect to be using the same hardware over the course of that decade? One of the biggest drawbacks to using on-prem hardware is keeping up with the upgrades. Leasing upgrade cycles are a thing for businesses, but for someone at home it seems like later in the decade the local system will be showing its age.
The 3090's I have would sell for more than new today... for the next couple of years or so it will be pretty bad on the hardware front. Until there is more RAM manufacturing capacity and NV has a credible competitor I don't think it will get better.
Heh, the hardware I bought I could easily sell for 2x so you may actually end up making money. Though I hope that that won't last.