Comment by whazor
1 day ago
But economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI.
However, I agree with the article that people will run big LLMs on their laptop N years down the line. Especially if hardware outgrows best-in-class LLM model requirements. If a phone could run a 512GB LLM model fast, you would want it.
Are you sure the subscription will still be affordable after the venture capital flood ends and the dumping stops?
100% yes.
The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!)
In some scenario where new investment stops flowing and some AI companies go bankrupt all that compute will be looking for a market.
Inference providers are already profitable so with cheaper hardware it will mean even cheaper AI systems.
You should probably disclose that you're a CTO at an AI startup, I had to click your bio to see that.
> The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!)
All going into the hands of a small group of people that will soon need to pay the piper.
That said, VC backed tech companies almost universally pull the rug once the money stops coming in. And historically those didn't have the trillions of dollars in future obligations that the current compute hardware oligopoly has. I can't see any universe where they don't start charging more, especially now that they've begun to make computers unaffordable for normal people.
And even past the bottom dollar cost, AI provides so many fun, new, unique ways for them to rug pull users. Maybe they start forcing users to smaller/quantized models. Maybe they start giving even the paying users ads. Maybe they start inserting propaganda/ads directly into the training data to make it more subtle. Maybe they just switch out models randomly or based on instantaneous hardware demand, giving users something even more unstable than LLMs already are. Maybe they'll charge based on semantic context (I see you're asking for help with your 2015 Ford Focus. Please subscribe to our 'Mechanic+' plan for $5/month or $25 for 24 hours). Maybe they charge more for API access. Maybe they'll charge to not train on your interactions.
I'll pass, thanks.
1 reply →
Datacenters full of GPU hosts aren't like dark fiber - they require massive ongoing expense, so the unit economics have to work really well. It is entirely possible that some overbuilt capacity will be left idle until it is obsolete.
3 replies →
> Inference providers are already profitable.
That surprises me, do you remember where you learned that?
1 reply →
> The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!)
which is funded by the dumping
when the bubble pops: these DCs are turned off and left to rot, and your capacity drops by a factor of 8192
1 reply →
Doesn't matter now. GP can revisit the math and buy some hardware once the subscription prices actually grow too high.
You have to remember that companies are kind of fungible in the sense that founders can close old companies and start new ones to walk away from bankruptcies in the old companies. When there's a bust and a lot of companies close up shop, because data centers were overbuilt, there's going to be a lot of GPUs being sold at firesale prices - imagine chips sold at $300k today being sold for $3k tomorrow to recoup a penny on the dollar. There's going to be a business model for someone buying those chips at $3k, then offering subscription prices at little more than the cost of electricity to keep the dumped GPUs running somewhere.
They will go down. Or the company will be gone.
Running an LLM locally means you never have to worry about how many tokens you've used, and also it allows for a lot of low latency interactions on smaller models that can run quickly.
I don't see why consumer hardware won't evolve to run more LLMs locally. It is a nice goal to strive for, which consumer hardware makers have been missing for a decade now. It is definitely achievable, especially if you just care about inference.
isnt this what all these NPUs are created for?
I haven’t seen an NPU that can compete with a GPU yet. Maybe for really small models, I’m still not sure where they are going with those.
any "it's cheaper to rent than to own" arguments can be (and must be) completely disregarded due to experience of the last decade
so stop it
> economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI
Uber is economical, too; but folks prefer to own cars, sometimes multiple.
And how there's market for all kinds of vanity cars, fast sportscars, expensive supercars... I imagine PCs & Laptops will have such a market, too: In probably less than a decade, may be a £20k laptop running a 671b+ LLM locally will be the norm among pros.
Paying $30-$70/day to commute is economical?
if you calculate depreciation and running costs on a new car in most places - I think it probably would be.
6 replies →
> Paying $30-$70/day to commute is economical?
When LLM use approaches this number, running one locally would be, yes. What you and other commentator seem to miss is, "Uber" is a stand-in for Cloud-based LLMs: Someone else builds and owns those servers, runs the LLMs, pays the electricity bills... while its users find it "economical" to rent it.
(btw, taxis are considered economical in parts of the world where owning cars is a luxury)
> Uber is economical, too
One time I took an Uber to work because my car broke down and was in the shop and the Uber driver (somewhat pointedly) made a comment that I must be really rich to commute to work via Uber because Ubers are so expensive
Most people don't realise the amount of money they spend per year on cars.