Comment by adastra22
13 hours ago
You got numbers? Because it seems perfectly possible to me. OpenAI and Anthropic’s marginal cost for inference is certainly far less than their API pricing.
13 hours ago
You got numbers? Because it seems perfectly possible to me. OpenAI and Anthropic’s marginal cost for inference is certainly far less than their API pricing.
See: https://www.wheresyoured.at/ He's been "numbering" for quite a while now.
Everything there is extremely speculative and I don't see anything that contradicts that inference itself could be profitable at massive scale. See https://youtu.be/xmkSf5IS-zw for example.
If the companies as a whole are destined to be profitable, or worth their valuations is a very different question. The only people who can truely answer that have time machines.
How can you say that with such certainty? You have no idea what it costs to run a 10T parameter model at extremely high concurrency.
These 1T param models running at <$3.00 per 1mm are certainly not profitable.
Because I’ve looked at what it would cost my company to self-host a SOTA sized model. For us it wasn’t worth it because the hardware is all bought up by frontier labs and we can’t get any supply. But if we could, at the prices they’re paying, it would pay for itself in 10-ish months. I assume further that they have economies of scale on top of what I was estimating.