Comment by tomp
4 days ago
I don't get this. It's not like the models are running on their own GPUs.
So if you're running open models on AWS GPUs, you might as well run Claude (which AWS supports, and doesn't share any data with Anthropic).
Same with Azure/OpenAI.
> (which AWS supports, and doesn't share any data with Anthropic)
I want to believe (pinky promises from terms of service don't count)
If you don't believe AWS follows their ToS, you shouldn't use any cloud provider for CPU/data storage/anything else either. That's not a mainstream position in the industry.
Sufficiently large enterprises could get their own GPUs to run the models though.
As long as you don't have "realtime" workloads, owning the GPUs quickly becomes the economical option. The main cost problems is in e.g. chat applications where the workload is spikey, and users expect an near-instant response, for which you need to scale the GPUs to the highest spikes of the workload.
You really don't need to be large. $100k can buy you a lot of compute and it's less than hiring an engineer. With that kind of money you can build an LLM server for a dozen people.
One engineer's salary to accelerate a team of twelve is so cheap you can't afford not to.
Open models on-prem is the future, not a single doubt in my mind.
I'm old enough to remember when my company had everything on-prem (both analytical and operational databases and servers) due to cost and security concerns. Nowadays we have everything on GCP.
The biggest problem we had with on-prem was maintenance as it took a lot of staff and time to ensure decent reliability.
> which AWS supports, and doesn't share any data with Anthropic
Ah... oh.
Well, it's a nice thought.