The difference being that an LLM request is not an operating system. Since they're compartmentalized and ephemeral, you can very easily distribute requests among your available hardware so that you can switch off machines during periods of low activity.
The difference being that an LLM request is not an operating system. Since they're compartmentalized and ephemeral, you can very easily distribute requests among your available hardware so that you can switch off machines during periods of low activity.
Your capital costs for buying those machines don't go away.
That's a problem that already exists in power generation and delivery, and it's already been solved. Bills are sums of fixed terms and variable terms.
3 replies →