Comment by visarga
2 hours ago
Local is never cheaper than cloud because they can do batch inference, and that means you load model weights once to produce 128 tokens on 128 sessions in parallel not 1 token on 1 session like local models. Local models rarely get to high utilization factor, they spend most of their time waiting.
If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.
> not 1 token on 1 session like local models.
Local models can absolutely run in batch, what are even talking about?
> If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.
Even if you ran sequentally, single session, a _finetuned_ tiny (8B) local model on narrow tasks would abolutely mog SOTAs, any of it - Fable, Opus, Sol you name it.