Comment by ketothekingdom

Definitely! It's more of an intuition but doesn't explain what vector gets put in the weights and the analysis doesn't involve the learning dynamics at all. I link some papers at the end that do think about how memory relates to the dataset. But the architecture having this storing+lookup property probably is important in helping transformers remember so much information.