Comment by Kuxe
5 days ago
A discovery by a giant is in some sense a new base vector in the space of discoveries. The interesting question is if a statistical machine can only perform a linear combination in the space of discoveries, or if a statistical machine can discover a new base vector in the space of discoveries.. whatever that is.
For sure we know modern LLMs and AIs are not constrained by anything particularly close to simple linear combinations, by virtue of their depth and non-linear activation functions.
But yes, it is not yet clear to what degree there can be (non-linear) extrapolation in the learned semantic spaces here.