Comment by pocksuppet

You always have an embedding matrix. The embedding matrix specifies which token vector is 'AL', which vector is 'AK', and so on, and which one is 'Cheyenne'. You could do it trivially and say that 'AL' is 1,0,0,0,0,..., 'AK' is 0,1,0,0,0,..., and 'Cheyenne' is ...,0,0,0,0,1, but that's still an embedding matrix - one which doesn't scale.

I didn't really follow the article to its conclusion. Surely the logical conclusion of "higher powers are better" is to take the limit as power goes to infinity - which is equivalent to just finding the most similar key and then returning the corresponding value. Why would you take a combination of all powers, when you know that higher powers work better?

Nothing in the article explained why we need "a memory" to work in this particular way that leads to it becoming a kind of transformer.

Talking about context tokens storing memories makes no sense. By definition, they are not memories, because they are all from the current prompt!

The whole blog post feels like a just-so story designed to make you think transformers were obvious.