← Back to context

Comment by ketothekingdom

1 day ago

Hey author here! I have a short summary here if people find it helpful: https://x.com/heyyalexwang/status/2111158818730397932?s=46

14 comments

ketothekingdom

Reply
rd11235  21 hours ago

This is one of the most well written blog posts I've seen in a while. Thanks

  • ketothekingdom  20 hours ago

    Thank you! It's been sitting in my head for so long. I'm glad I finally got to share it with people

    • 3abiton  1 hour ago

      It was a great read. I've been recording a lot of the ideas I had, and especially after discussions with people. Sadly never found time to sit down, digest them and write them down yet. Your article is quite inspiring!

plasticeagle  14 hours ago

I find some aspects of your writeup to be... quite annoying if I'm honest. The writing is good, but you fail to define several of your terms early on.

What is d_k - I assume it's the dimension of the k vector. Similarly, I assume d_v is the dimension of the v vector. What is the "i" suffix for here at this early stage? It doesn't need to be there, it's just confusing.

You say the keys are "used for lookup based on similarity". What does this mean? Similarity to what? The existing keys? Something else?

Your first example then introduces the identifier "M", which seems to be the number of memories. Again, this is not defined. It speaks again of similarities, and introduces "weights". But the matrix Mxd_k is a matrix made of all your keys, which aren't "weights" - or at least this term is new. The term "tokens" makes an appearance around this time also - something else that needs defining.

I think you need to go through the text of this article, with an eye to someone who is reading it for the first time. Define every term you use, remove unnecessary notation - and also perhaps explain at the start why one would be interested in a "memory store" of this nature.

There is a place for something that really explains what's going on here. I've yet to find it.

  • newsy-combi  32 minutes ago

    Agreed 100%. Mathematicians SUCK at writing. No matter how hard they try, they'll gloss over important non-trivial definitions. And I think this author has been trying... but seems to at the same time ignore the negative feedback like yet another academia snob.

  • justinnk  13 hours ago

    I found many of the things are clearly defined after the sentence „Here are the terms we’ll use throughout.“ right at the beginning (e.g. M, d_k, d_v).

    • newsy-combi  38 minutes ago

      Where he does NOT state what d_k or R^d_k mean and there are countless other examples. I happen to know because of classes I took, but it's insane and obnoxious to assume that every reader did or went through any higher education at all. This is not high school math. It's not terribly complicated either, making this situation even more frustrating. The author could have spent just a little extra effort explaining these things to multiply his target audience - but he's either incapable to think from a different POV or pandering to his peers by not wasting time defining "the obvious". Pure academic blindness on full display here.

  • jkman  9 hours ago

    Instead of being condescending and annoying yourself, did you even try to reference other texts or resources to try to figure those things out? This is a freshman's complaint

aanet  6 hours ago

Fantastic blog post.

Curious to hear more about the stack for your blog? Thanks