Comment by throwitaway222
21 hours ago
If companies are really doing this, then we're saying they have no problems spending tens of millions to get somewhat decent TPS and then having their employees complain they are timesliced and getting lots of timeouts because their org has 500 employees?
This is all spitballing, but I'd wager it's a third the type of workloads, a third hedging against your business depending on a single external provider, and a third trust.
Not everybody is coding or doing work that lends itself to burning tokens for warmth. Reuters for example seems to be more interested in using it for research, editing and formatting citations and the like. There's only so much of that work that needs doing, it doesn't always need to be real-time, and they probably don't see it scaling exponentially. They also need to be very aware and in control of their model's biases, or they risk it compromising their work output.
It's widely expected that all of the major providers will need to - and surely want to - drastically raise prices to justify the ludicrous amount of capital they're burning. Multiple companies have already talked about how their AI costs have exploded, and from what I understand that scale of enterprise is paying API rates. I would be disappointed if big business wasn't having a think about what that liability could look like. It's one thing to be reliant on a relatively "stable" vendor like Microsoft for Windows and Office, another to get AWS sticker shock, and then this is promising to be an order of magnitude worse.
Then just plain trust. What if ChatGPT starts recommending your competitors products, or the USA bars export of Anthropic's latest model (again, but for real this time), or they stop serving a model your business now depends on, and so on... That's a lot of risk to leave outside of your control.
Open weights != self hosted
At least accord to the MIT study last year, most employees are using their own AI subscriptions to do work.
Keep in mind that a vanishingly small number of workers are SWE's churning millions of tokens daily.
This is a very odd take. I burned something like 600 million tokens in a single week - a very highly productive week - but I can't imagine someone doing less than a million in a day unless they were truly sleeping.
I'm not sure why someone hasn't developed a company offering services that distributes AI across all idle or under-utilized VM's and PC's for enterprises in order to serve open sourced models. Outside of the electricity bill, there's no additional expenditure and you get the AI.
We've all seen the office spaces where there's 200 empty computers on a floor. Combined, it's something like 500 cores at ~3 Ghz each and around 3 TB of RAM. The networking is already there and software like exo already exists.
Vibe code that app and business. It’s a great idea like a kind of SETI@Home for business but I’m not sure the compute and latency will really be good enough unless you’re talking about a business with literally thousands of machines sitting half idle and always on, or always mostly on. i imagine it would take maybe 50-100 MacBook Pros running such a service to get to Claude level performance for one or two people.
Is there some mock input one could calculate to actually verify that? That many modern Mac Pros is no small potatoes when it comes to compute power. Someone must have a pre made formula anyone can use to do closer estimates?
1 reply →
I have not seen this. The one time I did it was after a major layoff. People use laptops and take them home. I'm not saying it's not viable but the scene you depict I think is outdated.