Comment by crediblejhj
8 hours ago
Great post. I'm a first-year CS student without much math knowledge (also poor English). I'd like to check whether I got the key idea right. There are layers and each layer stores key, value pairs (normalized in this article). If I query Q then Q will be compared with every stored key "Q@A + Q@B + Q@C + ... = value". If A is the answer then Q@A will be 1, others will be interferences. D dimension only has d orthogonal directions, so if I store more keys than the dimension, I can't avoid remaining interferences. What we want is reducing them by going through function F. And this article introduces heuristic capacity results with different memory store functions (linear -> ReLU -> self attention). Is it roughly right?
self attention is the best way to suppress interferences?
Before getting too hung up on memory and seeing these as key value stores, or self-attention is suppression of interference, recognize that a lot of experts and people in these areas disagree with this analogy for fairly deep reasons (I link other resources and discussion here: https://chatgpt.com/share/6ad3aa3d-d110-83ea-a8c5-8653b34743...