Comment by lazharichir
14 hours ago
> Put yourself in the user's shoes. On what basis should one pick a model for a given task?
Create a realistic eval suite for your use case using your actual data that you manually label, with easier, more complex, rarer samples, and run each model you are considering against it and decide based on the usual confusion matrix, cost, latency, or whatever metrics matter to you. There's really no other way to know – and you might be surprised that cheaper models can often perform your task(s) relatively well at a fraction of the cost.
> Create a realistic eval suite for your use case using your actual data that you manually label, with easier, more complex, rarer samples, and run each model you are considering against it and decide based on the usual confusion matrix, cost, latency, or whatever metrics matter to you.
I’d imagine there’s only a tiny percentage of users that can or want to do this.
if you consider every single person that uses some LLM then sure, it is tiny percentage, but I would wage that large percentage don't write technical documentation as GP. large userbase probably don't need THE best solution for their use case and will be comfortable with A solution that is within their budget. so for them simple luna < terra < sol < astra would suffice.
but if you want to use some LLM agent for anything commercial and you want to get the best and at the same time you don't want to overpay the easiest and the surest would be to create your own benchmark for your specific use case.