Comment by NorwegianDude
6 days ago
A common extreme misconception is that inference is expensive and that providers are loosing a lot of money. Inference is extremely lucrative and profitable.
6 days ago
A common extreme misconception is that inference is expensive and that providers are loosing a lot of money. Inference is extremely lucrative and profitable.
Inference is the phase where they make money. But the question is whether they can be profitable overall as training continues to balloon.
I think the case for this is pretty strong actually. Last year my company was maybe willing to pay $100 a month to Anthropic (per developer). Today we're all on the $300 plan without any hesitation. If Fable ever becomes available as the default model, I imagine my company would be willing to pay in the $500-$1000 range per month per developer.
Okay, but that still has a limit, right? Do training costs have a limit? Everyone is in the frothy stage of this technology wave and they continue to buy more, but training the next model requires exponential increases in model sizes to get the same sorts of model performance increases, which suggests exponential cost increases, too (even ignoring temporary cost factors such as RAM price increases). You say your company will double or triple what they are paying today; how far are they willing to go? At some point they are going to have to cut developers to fund it (e.g., cut half the developers and give the survivors each an AI assistant with $180k in token budget, captured from the salary savings), but that also presupposes the productivity gains are there to support it.
1 reply →