Comment by afavour
5 hours ago
Wouldn’t you just have fallbacks? Today’s frontier models are just better than the other models, they don’t really have a ton of entirely unique abilities that can’t be replicated with more time and effort.
So you use the frontier model, then when you can’t you accept things are less efficient. The alternative (right now) is to be less efficient all the time, I don’t see any advantage to that.
Yeah. It’s not the end of the world.
But, it is a big own goal, because once you invest in building evals for your internal use-case, 1) it’s easier to switch your model to whatever is cheapest, and 2) it’s way easier to fine-tune an oss model.
Evals are annoying to build and most companies were fine to rest on vibes. Now many companies have to do the work for insurance.