Comment by tossandthrow
4 hours ago
I don't think that is true.
I think more of the capabilities comes from the cross domains knowledge.
Anyways, software is also designed for a domain. The reason why it is so adept at making a fitness tracker is likely because it knows about biology.
Sure, that's a plausible theory but I haven't seen that anyone's proved it.
An alternate theory is that models need lots of input knowledge to learn complex reasoning, but don't need so much at inference time. An example is arithmetic. Early in their training, models do arithmetic (poorly) by pattern-matching on memorized examples. Eventually they grok arithmetic and stop pattern matching, and then they don't use the examples anymore.
The fitness tracker doesn't take much knowledge of biology. Large models have quite a lot of biology knowledge that most people will never use. Same goes for lots of other topics. For basic knowledge that easily fits in context it could search the internet, or a local collection of introductory textbooks.