Comment by antasvara
14 hours ago
It's a few things I assume:
1. Inference providers are largely a commodity business. While demand outstrips supply, there are many providers of that supply, which does somewhat constrain how much leverage a provider has.
2. The models are not interchangeable as of yet (though this seems to be changing). A lot of people want the latest Anthropic or OpenAI model; as such, the labs still have considerable leverage.
3. A lot of the inference providers have used their leverage by doing equity deals with the labs, agreements to have some of the deal revenue spent on compute, etc.
What's the evidence that demand outstrips supply? Is there a wait to get an inference provider or interruptions in availability?
GPU prices are sitting well above MSRP, even for older generation cards. Datacenter growth is also being pushed pretty hard by most of the cloud providers. More anecdotally, the model providers I use have been "throttling" my requests (usually by bumping them down to a lower model version) more frequently. Granted, that could be for profit reasons, but I think it all generally points to a lack of capacity.
>Is there a wait to get an inference provider or interruptions in availability?
I think that the inference providers and model vendors are mostly handling it behind the scenes with more aggressive quantization, increasing API prices, deprecating more generous monthly tiers, etc.
this info is dated