Comment by jmalicki

4 hours ago

> Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right?

I see arguments like this all of the time in ML. There is a large branch of statistical theory that can cast some doubt on this.

Stein's paradox: when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately.

This goes deeply against "common sense", but is mathematically proven, empirically proven, etc.

If you are just taking a chess model vs. an LLM, you are right.

But if you want a chess model, a medical diagnosis model, and customer service chat model, at a high level the intuition of Stein's paradox is that one combined model will get you better performance than three task-specific models.

Most practitioners don't get this because what amounts to graduate-level stats theory is often brushed aside, and we keep relearning it the hard way.