Comment by Marshferm

Quoting Timos Moraitis a Neuromorphic PhD

"For reasons like this, "in-context learning" is not an accurate term for transformers. It's projection and storage, nothing is learnt.

This new paper has attracted a lot of interest, and it's nice that it proves things formally and empirically, but it looks like people are surprised by it, even though it was clear."

https://x.com/timos_m/status/1983625714202010111