Comment by Aurornis
18 hours ago
> as a clear sign they are scared shitless of local models,
The number of people with hardware to run a model like that is very small.
It's going to be many years before hardware to run medium size local models is widely available. Running mid-size local models doesn't even register as a factor in pricing decisions right now.
It's such a small part of the market and will be for many years, until RAM and other hardware prices come down.
> The number of people with hardware to run a model like that is very small.
True, but the number of people who pay the non-subsidized API price is also very small, and it's all in big companies who would have no issue buying or renting hardware for that purpose.
Ram prices suggest there is going to be a mighty glut of the stuff in a couple of years. We may easily see 32->64->128 GB system RAM off the back of that and CPUs seem to be moving to unifying VRAM and main memory.
I don't think running a ~200B parameter model is that out of reach, it might be usual by the mid 2030s for people in the tech world. Factor in the crazy improvements in quality given a number of parameters too.
But why? HBM is not reusable by consumers, and we would probably be at ddr6 by the time used ddr5 comes crashing down.
but the number of small to medium sized companies which really want to run (some of) the AI usage under their control and are willing to drop some 100ks on it aren't that few (especial for non US companies given the state the US is in)
> but the number of small to medium sized companies which really want to run (some of) the AI usage under their control and are willing to drop some 100ks on it aren't that few
To this I'll add companies who don't mind paying for a non-US service provider to deliver the compute required to run local models.
Just like some users are happy to run GPT6 Luna, some users are happy to run gemma4 et al.
> The number of people with hardware to run a model like that is very small.
But there are a lot of companies who do and rent it out as a service.
Idk, I’m running 180b on $2500 of hardware at 120 T/S (single user of course) but I bought it before the clown show. Now it’s probably more like $10k lol
I expect I’ll be able to run 1/2 terabyte models in 5 years for a couple $K in hardware. Inference is going to be a lot like Bitcoin mining. Only the peak of the power/performance curve is going to be profitable. Which means a glut of surplus hardware that can’t make money but makes tons of tokens.
I hope you are right. I've been waiting for GPU prices to ease off ever since ASIC mining took off.
Many years is only long in relation to the AI hype cycle. In terms of a human lifespan it is not that much, maybe 10 years at most and that's pessimistic. We technically live in that future already it is just very unevenly distributed.
A human lifespan doesn't have that many decades in it either...
> The number of people with hardware to run a model like that is very small.
That's not the problem anyone is facing, and analysis anyone is doing. At all.
The problem is the huge AI-related budget that companies who dove head first onto AI are facing.
Therefore the problem everyone is faced with boils down to whether they keep paying the likes of OpenAI a de-facto business tax, or they replace that tax with a one-off CapEx invoice followed by a utilities bill.
And you don't need a frontier model to do most things. In fact, the biggest challenge in using local models in agentic coding is the models supporting these agent-based tools.
> It's going to be many years before hardware to run medium size local models is widely available.
Nonsense. Speaking for myself, my employer already introduced a hard cap on AI credits while rolling out hardware support for local models, and even went to the extents of rolling out workstations with 64GB of RAM as a baseline. Consequently, we are using the likes of gemma4 for a fair share of everyday tasks. This has a transformative effect on your workflows because suddenly you don't ration tokens, and agent orchestration doesn't become a cash guzzler. So far the only constraints are tool support and token throughput, but that's not a concern.
I wonder how this compares to a hard(-ish) cap on Sonnet-and-better models, and unlimited access to Haiku. That’s one of the policies we’re considering.
[dead]
The average person may not even own a personal computer, has a company-owned notebook with barely 16 GB of RAM and maybe a TPU, and here you are saying that owning a prosumer GPU and more RAM than what most people will buy across their entire lifetimes is common.
The average person doesn't have an interest in AI either.
I'm talking small and medium businesses, who have productive gains using AI, where a 3k€ PC is common. We have many.
> has a company-owned notebook with barely 16 GB of RAM
Mind you, some local models run on 16GB of RAM, with some headroom to do development work.
> Not completely obvious.
I'd have to spend what, 5-10k today to run flash models? It can't be justified as long as I have Luna on 20$ sub. I'd have to be doing so much heavy unsupervised work for the economy to make sense.
The number of people who have 192GB RAM at their disposal is minuscule because there was no reason to have that much just recently unless: 1. You're a professional in some compute heavy field AND 2. You don't use a cloud or in-house enterprise infrastructure
If you think that 192GB of VRAM on a video card that costs 3k€ is "pedestrian", I think you're a bit out of touch with the average consumer.
12 months ago buying 192 gigs of ram was pedestrian? Really?
Just checked, 192GB of DDR5 was 500€ vat excl. That's what we paid per for each set of 4x48.
500€ for a production tool seems fairly innocuous. We're not in the 10k€+ workstation territory here. At least we were not.