Comment by forrestthewoods
10 hours ago
Inference costs are higher than training now. I think.
Nvidia is king of general purpose training chips. But inferences can be specialized.
10 hours ago
Inference costs are higher than training now. I think.
Nvidia is king of general purpose training chips. But inferences can be specialized.
What makes you think this? With wider adoption the ratio shall shift in favor of inference. And API price is becoming more important than SOTA capability.
> With wider adoption the ratio shall shift in favor of inference
Yes? That’s why more money will be spent on inference than training?
I’m talking absolute cost. As the number of people using AI and burning tokens goes up the amount of spend on inference goes up.
I am fairly confident that Anthropic has way way more GPUs serving Claude Code to users than they have training models. They’ve got a lot of users!!
> API price is becoming more important than SOTA capability.
Also yes? This is why custom silicon for efficient inference makes sense!
I think we’re in total agreement here :)