Comment by tempay
16 hours ago
Are they? I only ever see unsubstantiated claims for this whereas I see many justifications that interference is comfortably profitable in isolation.
16 hours ago
Are they? I only ever see unsubstantiated claims for this whereas I see many justifications that interference is comfortably profitable in isolation.
SpaceX's has disclosed that they're loosing $2Bln a quarter on A.I - and rising - in their IPO documents.
Anthropic told the Department of War-nee-Defence that they'd made $5bln total, which is a lot LOT less than what they're spending.
We'll see what's in OpenAi's IPO later this year I guess. I'll be very surprised if they're losing less that $100bln a year.
Is it capex of training new models and hiring people for 250mln pay packages? Or is it opex running inference?
Its basic math, go calculate max sessions for a certain tps on any hardware. Session# * tps * 86400 (secs in a day) * 30 days.
You'll realize real quick its not profitible. You cant just say things you don't like to hear are unsubstantiated without verifying.
Not to mention, subscriptions.. $2mm in GPUs being given out for 5 hrs a day at a cost of $200 a month.
I could easily say that everyone who says its profitible is msking unsubstantiated claims lol.
>Its basic math
Yes, once you have modeled the problem correctly and you know all the input parameters. This is not that: Session# * tps * 86400 (secs in a day) * 30 days.
I don't think there is enough public information to check Anthropic's claims regarding inference profitability. It depends not just on unknown technical factors but also on agreements they have with other companies.
I agree that we dont know how expensive SOTA is. But yes my math should give you the max amount of tokens you can sell per month, and its not remotely profitible for most of the larger open source models (at their current pricing). Im not sure why a 10x larger model that is more in demand would be profitible when its only 5x the price.
Its possible you could pay off hardware for Kimi 2.6 after maybe 2-3 yrs (by providing low tps / high concurrency) but you're now out of warranty and have been running your machines full throttle 24/7 for 2-3 years.
This is why moonshot attempted to double the price when they released 2.6 but then it got driven down by North American capital subsidies.
We should specify which subscription plan we are talking about. You seem to be talking about the Anthropic Claude Max plan. I think it's consensus that these flat rate type of subscriptions are loss leaders, as they come with restrictions how you can use the API via T&C, namely only with Claude Code et al. They are meant to hook developers into their products.
Shouldn't we compare the API pricing, where we pay per token? The whole point of local inference is that we don't have any restrictions regarding product use or time limits, so it would only be fair if we compare it to a plan that offers the same. And even that is only a first approximation, because the commercial models are usually much more capable than the open weight models.
> I could easily say that everyone who says its profitible is msking unsubstantiated claims lol.
And people who don't understand the difference between capex and opex are making uneducated claims. It's not basic math.
Running an inference data center is a mix of variable and fixed costs. The fixed costs are currently in the billions of billions of dollars for pretty much any investment in this space. Many of those fixed costs have (currently) unknown refresh cycles. So, unless you have access to the financial books of these companies it's currently just speculation whether inference is profitable.
You got numbers? Because it seems perfectly possible to me. OpenAI and Anthropic’s marginal cost for inference is certainly far less than their API pricing.
See: https://www.wheresyoured.at/ He's been "numbering" for quite a while now.
1 reply →
How can you say that with such certainty? You have no idea what it costs to run a 10T parameter model at extremely high concurrency.
These 1T param models running at <$3.00 per 1mm are certainly not profitable.
1 reply →
To some degree I think there's a hope that it becomes like a gym membership. If everybody used their membership, the gym would be too crowded. It's all of those memberships that people feel like they need to have but don't use where the extra profit comes in.
As long as the power users are paying per token, everything is good.
Really? This is what we expect from this amazing world changing technology? People will sign up for it and not use it? Good business plan, how can I invest? /s
1 reply →
Especially since their costs might be multi-year investments. It's too early to judge the quality of those investments.