Comment by zardo

GPU prices are sitting well above MSRP, even for older generation cards. Datacenter growth is also being pushed pretty hard by most of the cloud providers. More anecdotally, the model providers I use have been "throttling" my requests (usually by bumping them down to a lower model version) more frequently. Granted, that could be for profit reasons, but I think it all generally points to a lack of capacity.

>Is there a wait to get an inference provider or interruptions in availability?

I think that the inference providers and model vendors are mostly handling it behind the scenes with more aggressive quantization, increasing API prices, deprecating more generous monthly tiers, etc.