Comment by janalsncm

On narrow domains, it is very common for small models to match or outperform larger ones at a fraction of the parameter count.

For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.

https://arxiv.org/pdf/2311.09205

In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.