Comment by giraffe_lady
14 hours ago
Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models.
The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.
And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.
> claude fable is better at writing both code and prose than smaller code- and prose-specific models.
There's a million ways to define "better at writing prose", and it's only better at some of them. It's worse at many others.
On narrow domains, it is very common for small models to match or outperform larger ones at a fraction of the parameter count.
For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.
https://arxiv.org/pdf/2311.09205
In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.
I think the context here is that small models run locally, not rented from a cloud.
Yes small models are and will be useful for lots of stuff for several reasons.
But the idea they’d be better than a bigger model is cope, you’re pretty much always better off running the biggest one you can bring to bear within your constraints.
Fable 5 is actually a lousy writer. Opus 4.6 is the best for writing and prose assessment. Gemini 3 is smarter at reading comprehension but tends to be more unstable in judgement.