Comment by int_19h
5 days ago
At the moment, the only way you can tell if the model is good for a particular task is by trying it at that task. Gut feel is how you pick the models to test first, and that is also based largely on past experience and educated guesses as to what strengths translate between tasks.
You should also remember that there's no free lunch. If you see models below a certain size fail consistently, don't expect a model that is even smaller to somehow magically succeed, no matter how much pixie dust the developer advertises.
To some extent there must be a free lunch, because today's 30B models are enormously better than the 30B models that existed a year ago.
I suppose it's an open question whether there is another free lunch or whether the 30B models in a year will be not much better than our current ones.