Thank you for putting that excellent article together
The concepts of fidelity/recall and the images/animation at the top reminded me of this small playful project that touches upon these topics: https://nicobrenner.github.io/lossy-git/
and others.
Hippocampal sparse coding/orthoganalization/pattern separation and completion/etc may be interesting search terms for you, if you are unfamiliar. If you do happen to be a hippocampal nut like I used to be during grad school, I salute you
I find some aspects of your writeup to be... quite annoying if I'm honest. The writing is good, but you fail to define several of your terms early on.
What is d_k - I assume it's the dimension of the k vector. Similarly, I assume d_v is the dimension of the v vector. What is the "i" suffix for here at this early stage? It doesn't need to be there, it's just confusing.
You say the keys are "used for lookup based on similarity". What does this mean? Similarity to what? The existing keys? Something else?
Your first example then introduces the identifier "M", which seems to be the number of memories. Again, this is not defined. It speaks again of similarities, and introduces "weights". But the matrix Mxd_k is a matrix made of all your keys, which aren't "weights" - or at least this term is new. The term "tokens" makes an appearance around this time also - something else that needs defining.
I think you need to go through the text of this article, with an eye to someone who is reading it for the first time. Define every term you use, remove unnecessary notation - and also perhaps explain at the start why one would be interested in a "memory store" of this nature.
There is a place for something that really explains what's going on here. I've yet to find it.
I found many of the things are clearly defined after the sentence „Here are the terms we’ll use throughout.“ right at the beginning (e.g. M, d_k, d_v).
Instead of being condescending and annoying yourself, did you even try to reference other texts or resources to try to figure those things out? This is a freshman's complaint
I think I can follow most of this, or probably will be able to do so with some help from Claude as we go through it with my questions, but I thought I'd ask humans first.
I've watched most of Karpathy's videos but it's been a while so if I ask a dumb question it's because I'm at a very basic level from a while ago.
To try to turn this into something concrete, let's say I want to store as facts the 50 state capitals. We'll do the most basic tokenization possible, so I have ['AL': 1, 'AK': 2, 'AZ': 3, ... 'WI': 49, 'WY': 50, 'Montgomery': 51, 'Juneau': 52, ... 'Cheyenne': 100]
(Do I need an embedding matrix here? I feel like I need a 100x64 embedding matrix, one for each "token", even for those very first linear examples? Maybe I need two 50x64 matrices?)
But then I'm learning randomly initialized V and K matrices by going through and dotting my 'AL' row by K and my 'Montgomery' row by V and then dotting them together, and going through all of the rest of the states similarly and then optimizing a loss function until I get a V and K that gives me good performance?
I kind of wanted to go one step further and adding two more tokens, a 'StateHasCapital' token and a 'IsCapitalOf' token (say 101 and 102), and somehow use that to learn with compound keys ['StateHasCapital'+'AL': 'Montgomery'] - if I'm really learning facts, can I train on all 50 States->Capitals and maybe 25 Capitals->States and then as a test to see if I'm learning "facts", use the remaining capitals that I didn't train on to see if I can look up what state has the capital city of Olympia?
I might be conflating a whole bunch of steps that don't make sense in that initial linear example and adding complexity, and maybe trying to fit too much into the later examples too (I don't see the 'Query' matrix in the any of the parts of the post that get into self-attention, and maybe this post is just about "here's how memory works, not how LLMs work"
The 'you' this article--and other articles like this--mentions, doesn't definitely include you, nor me, nor people like us.
Recently, Jensen Huang reportedly said he doesn't know where he lives (the address) with a straight face. This 'you' situation here is kind of like the Jensen situation: neither party understands what it's like to be the other party.
This is actually useful. I didn’t know that generic “you” meant “one”. I always interpreted it as “you the reader” and felt desperate once a week when reading articles like these.
My guess is comment meant that in the context of the headline "you" normally would refer to the average Joe on the street; in this case, it means the average Joe who has a PhD in math, CS, or whatever. It all certainly went over my head and probably 95% (or slightly less than that given this is HN) of the people who clicked the link. But I would take the headline as a compliment that I don't deserve.
"Here are the terms we’ll use throughout. A memory store is a function $f$ that holds $M$ memories, for $i = 1, \ldots, M$. A memory is one key-value pair: a key $k_i \in \reals^{d_k}$, used for lookup based on similarity, and a value $v_i \in \reals^{d_v}$, which is what we want to retrieve. To use $f$, we query it with an input $x$, and it recalls an output $f(x)$. A good memory store recalls $v_i$ when we query it with $k_i$, or with another vector similar to $k_i$. In the literature, a memory store that recalls by similarity is called an associative memory."
And thought "I need to disable noscript because I get crappy page view". I disabled and it didn't help. It's a little sad that there is no explanation in the article what all that dollar signs mean. I feel excluded
Thats MathJax, you need to allow JS to load from cdn.jsdelivr.net. If doing that doesn't cause the math to render, you likely have some other issue (perhaps a very old / strange browser, a caching issue, or some other kind of blocker additionally preventing the JS from loading).
Great post. I'm a first-year CS student without much math knowledge (also poor English). I'd like to check whether I got the key idea right.
There are layers and each layer stores key, value pairs (normalized in this article).
If I query Q then Q will be compared with every stored key "Q@A + Q@B + Q@C + ... = value".
If A is the answer then Q@A will be 1, others will be interferences.
D dimension only has d orthogonal directions, so if I store more keys than the dimension, I can't avoid remaining interferences.
What we want is reducing them by going through function F.
And this article introduces heuristic capacity results with different memory store functions (linear -> ReLU -> self attention).
Is it roughly right?
self attention is the best way to suppress interferences?
Before getting too hung up on memory and seeing these as key value stores, or self-attention is suppression of interference, recognize that a lot of experts and people in these areas disagree with this analogy for fairly deep reasons (I link other resources and discussion here: https://chatgpt.com/share/6ad3aa3d-d110-83ea-a8c5-8653b34743...
This is a really interesting article! While I can’t profess to understand all the math it is a great way to demonstrate it.
I recall awhile back hearing someone ask whether intelligence was actually the same or similar to “compression” or storing all of this knowledge in weights - IE the ability to create something from reason or weights or relationships / connections. This article makes it even more interesting!
Funny how age shows up in some cases. For me the first thing that came to mind, when starting to read the article, was Ranulph Glanville's 1978: "What is memory that it can remember what it is?" (Progress in cybernetics and systems research - vol 4)
Not sure if this was a joke or a timeline glitch, but the year was 1976 and the volume was 7 ;)
> Glanville, R. (1976). What is memory, that it can remember what it is? In R. Trappl et al. (Eds.), Recent Progress in Cybernetics & Systems Research, Volume 7 (Proceedings 3rd European Meeting on Cybernetics and Systems Research). Washington, DC: Hemisphere Press.
"interference-to-signal ratio", "capacity", ... These terms, and their context, are things a communications/electrical engineer would understand. I encourage comms engineers to start learning how AI works under the hood. Chances are, a lot of what you know about maximising channel capacity and throughput is applicable to AI (can be seen as a channel with memory?).
Great article. My favorite fact about "transformers as a hopfield network" is that transformers are so good at memorizing that they only need one step to find a suitable memory as opposed to multiple steps for a vanilla hopfield network
In an attention layer, the query, key, and value are all computed at inference time using weight matrices that are determined as a result of the training. If X is the token representations entering the attention layer, then each of Q, K, and V would be determined by multiplying X by the corresponding learned weight matrix for queries, keys, and values:
Q = XW_q
K = XW_k
V = XW_v
From there, you can think of K and V like keys and values in a Python dictionary, and Q as the key you pass (e.g. d[Q] in Python to return some value V). The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
> The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
I understand you meant to compare it to dicts, but this is more like (the only?) similarity to outer product memory. Worth calling out since GP is confused about this connection specifically.
To clarify slightly, K and V are used to construct the dictionary, and then Q is the lookup key D[Q] as described (with the lookup being based on similarity rather than exact hash as it would be in an actual dictionary).
See my comments and links below, you are not wrong, IMO this is not actually a great analogy for understanding LLMs (other than as an explanation for why a trained one can so efficiently compress such a large amount of information).
So refreshing to see actual human dialog at the start of an AI post/paper/explainer, instead of having that visceral "I'm just reading AI slop" feeling. Even if it's totally unrelated, the little bit about playing the Dune board game made me much more engaged and willing to listen to what the rest of the content had to say.
Goodness, someone posts a great article which IMO accessible math, and people get upset.
If you think that is crazy, ohh boy, just wait til you see all the other papers underlying LLMs, AI, and optimization in general.
I appreciate this post because starting from linear algebra is the correct way to form an intuition of the higher dimensional semantic spaces we're dealing with.
Yeah, the notation and math concepts also all just first-year calculus (or analysis, more accurately) and stats (and the latter being only E for expectation). Literally just summation notation, blackboard bold R for the real numbers, double bars for norm / distance, and then max/min with subscript for the quantity being minimized.
Embarrassing to complain about notation so simple, and, frankly, if you can't understand math concepts and notation this basic, you simply can't meaningfully understand modern neural networks: without the math, all you are doing is substituting loose analogies and metaphor for understanding. This is clearly an article for people that want more than that.
This article does two things which make it unhelpful for me:
1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters?
2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix?
To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings:
> the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning
What made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset.
So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.
> This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters
At the risk of stating the obvious, this is a forum where people can post arbitrary links to other sites, some of which might have audiences that might not be "the exact set of people who read Hacker News". I'm not sure you'd be able to make a compelling argument that alexwang.ai is aimed at software developers and no one else.
Yes, but you need to consider your audience when posting links to this site. For example, if I posted to HN a link to a really good cookie recipe, it wouldn't be a good fit for this site. The author did nothing wrong, but that doesn't make it a good submission to HN.
This article is a bit more of a grey area. It is on a subject that is deeply relevant to this site and I'm sure some HN readers will get value from it. My point is that an even better submission to HN would have been an article on the same topic but written for programmers rather than mathematicians.
FWIW, I found the idea of interference-to-signal ratio as a way to think about memory capacity to be intuitive and was really interested in getting a better sense of how models increase capacity, until it hit the first math notation formula. My disconnect was based on the approachable tone of the intro, from which I assumed it would be a more generally accessible article.
To be clear, I have no problem with math people conveying math stuff with math notation but it seemed like the type of article which, even if it will get into notation, first at least explains "the main idea of it" without notation, then expands with notation as necessary when fleshing details or implementation.
That is not an accurate read of my post. My post was about two things:
1. The notation used
2. The lack of concrete examples
I'm not saying they should remove any of the content, just put it in a form more accessible to us. For example, I can read and mostly understand a news article in French, but it will take a lot of time and effort. If it was written in English, it would be easy for me to read. The two news articles have exactly the same content, one is just written in a form easier for me to understand.
Well putting this article in math notation without defining any of the data structures behind it, that's not like putting the article in French, it's like putting the article in Japanese. I can't understand any of it. Some Japanese speakers on this site are saying "I don't understand the problem, just learn Japanese." which is pretty much the least constructive comment they could make.
I guess you're not the target audience for this article then.
This is pretty approachable maths, and explain really well. I've seen far worse in programming language theory papers, which would qualify as "software development" in my mind. There are plenty of users of this forum that would find this paper useful and interesting, with sufficient math background to read it to boot.
What made it not useful to you likely makes it much more useful for me.
Fortunately, LLMs can work wonders in adapting content to be more accessible to a wide variety of audiences. Just give it a link and feed your exact post to it. It will be able to translate it into pseudocode and provide examples.
Well, you could tell your AI of choice to read the article and explain the math. I did that with ChatGPT and it helpfully translated a lot of it into Python for me.
>It's easy to forget that the average person probably only knows the formulas for Olivine and one or two Feldspars
That being said the llm is indeed the universal translator — the babelfish supreme — and is excellent at translating whatever you want into pseudocode in your language of choice.
Hey author here! I have a short summary here if people find it helpful: https://x.com/heyyalexwang/status/2111158818730397932?s=46
This is one of the most well written blog posts I've seen in a while. Thanks
Thank you! It's been sitting in my head for so long. I'm glad I finally got to share it with people
Thank you for putting that excellent article together
The concepts of fidelity/recall and the images/animation at the top reminded me of this small playful project that touches upon these topics: https://nicobrenner.github.io/lossy-git/
That's such a neat idea! Now I want to try it out too
https://www.nature.com/articles/s41586-024-08548-w
and others. Hippocampal sparse coding/orthoganalization/pattern separation and completion/etc may be interesting search terms for you, if you are unfamiliar. If you do happen to be a hippocampal nut like I used to be during grad school, I salute you
No, after many years, I still hate Twitter/X.
Please write a proper Blog post.
I find some aspects of your writeup to be... quite annoying if I'm honest. The writing is good, but you fail to define several of your terms early on.
What is d_k - I assume it's the dimension of the k vector. Similarly, I assume d_v is the dimension of the v vector. What is the "i" suffix for here at this early stage? It doesn't need to be there, it's just confusing.
You say the keys are "used for lookup based on similarity". What does this mean? Similarity to what? The existing keys? Something else?
Your first example then introduces the identifier "M", which seems to be the number of memories. Again, this is not defined. It speaks again of similarities, and introduces "weights". But the matrix Mxd_k is a matrix made of all your keys, which aren't "weights" - or at least this term is new. The term "tokens" makes an appearance around this time also - something else that needs defining.
I think you need to go through the text of this article, with an eye to someone who is reading it for the first time. Define every term you use, remove unnecessary notation - and also perhaps explain at the start why one would be interested in a "memory store" of this nature.
There is a place for something that really explains what's going on here. I've yet to find it.
I found many of the things are clearly defined after the sentence „Here are the terms we’ll use throughout.“ right at the beginning (e.g. M, d_k, d_v).
Instead of being condescending and annoying yourself, did you even try to reference other texts or resources to try to figure those things out? This is a freshman's complaint
Fantastic blog post.
Curious to hear more about the stack for your blog? Thanks
Really awesome write-up, and the kea is adorable!
I think I can follow most of this, or probably will be able to do so with some help from Claude as we go through it with my questions, but I thought I'd ask humans first.
I've watched most of Karpathy's videos but it's been a while so if I ask a dumb question it's because I'm at a very basic level from a while ago.
To try to turn this into something concrete, let's say I want to store as facts the 50 state capitals. We'll do the most basic tokenization possible, so I have ['AL': 1, 'AK': 2, 'AZ': 3, ... 'WI': 49, 'WY': 50, 'Montgomery': 51, 'Juneau': 52, ... 'Cheyenne': 100]
(Do I need an embedding matrix here? I feel like I need a 100x64 embedding matrix, one for each "token", even for those very first linear examples? Maybe I need two 50x64 matrices?)
But then I'm learning randomly initialized V and K matrices by going through and dotting my 'AL' row by K and my 'Montgomery' row by V and then dotting them together, and going through all of the rest of the states similarly and then optimizing a loss function until I get a V and K that gives me good performance?
I kind of wanted to go one step further and adding two more tokens, a 'StateHasCapital' token and a 'IsCapitalOf' token (say 101 and 102), and somehow use that to learn with compound keys ['StateHasCapital'+'AL': 'Montgomery'] - if I'm really learning facts, can I train on all 50 States->Capitals and maybe 25 Capitals->States and then as a test to see if I'm learning "facts", use the remaining capitals that I didn't train on to see if I can look up what state has the capital city of Olympia?
I might be conflating a whole bunch of steps that don't make sense in that initial linear example and adding complexity, and maybe trying to fit too much into the later examples too (I don't see the 'Query' matrix in the any of the parts of the post that get into self-attention, and maybe this post is just about "here's how memory works, not how LLMs work"
Would be nice if articles defined the requirements to be a “you”.
The 'you' this article--and other articles like this--mentions, doesn't definitely include you, nor me, nor people like us.
Recently, Jensen Huang reportedly said he doesn't know where he lives (the address) with a straight face. This 'you' situation here is kind of like the Jensen situation: neither party understands what it's like to be the other party.
https://en.wikipedia.org/wiki/Generic_you
This is actually useful. I didn’t know that generic “you” meant “one”. I always interpreted it as “you the reader” and felt desperate once a week when reading articles like these.
12 replies →
That said, the audience for this is still pretty niche.
(Even the audience for, say, 3b1b's explanation of what a transformer is, is niche.)
1 reply →
My guess is comment meant that in the context of the headline "you" normally would refer to the average Joe on the street; in this case, it means the average Joe who has a PhD in math, CS, or whatever. It all certainly went over my head and probably 95% (or slightly less than that given this is HN) of the people who clicked the link. But I would take the headline as a compliment that I don't deserve.
1 reply →
They're asking what prerequisites would be needed to understand the article.
1 reply →
"You could have invented..."
Oh you know there is gonna be maths in this one. (clicks link)
"Here are the terms we will use throughout: a 'memory store' is a function f..."
Yes. Ha ha! Yes!
Were you hoping they would explain what is essentially a rather sophisticated math structure without using any... math?
why?
Because I'm an idiot.
1 reply →
So I saw this:
"Here are the terms we’ll use throughout. A memory store is a function $f$ that holds $M$ memories, for $i = 1, \ldots, M$. A memory is one key-value pair: a key $k_i \in \reals^{d_k}$, used for lookup based on similarity, and a value $v_i \in \reals^{d_v}$, which is what we want to retrieve. To use $f$, we query it with an input $x$, and it recalls an output $f(x)$. A good memory store recalls $v_i$ when we query it with $k_i$, or with another vector similar to $k_i$. In the literature, a memory store that recalls by similarity is called an associative memory."
And thought "I need to disable noscript because I get crappy page view". I disabled and it didn't help. It's a little sad that there is no explanation in the article what all that dollar signs mean. I feel excluded
Thats MathJax, you need to allow JS to load from cdn.jsdelivr.net. If doing that doesn't cause the math to render, you likely have some other issue (perhaps a very old / strange browser, a caching issue, or some other kind of blocker additionally preventing the JS from loading).
You may have some other issue. That stuff renders as mathematical notation.
[dead]
Great post. I'm a first-year CS student without much math knowledge (also poor English). I'd like to check whether I got the key idea right. There are layers and each layer stores key, value pairs (normalized in this article). If I query Q then Q will be compared with every stored key "Q@A + Q@B + Q@C + ... = value". If A is the answer then Q@A will be 1, others will be interferences. D dimension only has d orthogonal directions, so if I store more keys than the dimension, I can't avoid remaining interferences. What we want is reducing them by going through function F. And this article introduces heuristic capacity results with different memory store functions (linear -> ReLU -> self attention). Is it roughly right?
self attention is the best way to suppress interferences?
Before getting too hung up on memory and seeing these as key value stores, or self-attention is suppression of interference, recognize that a lot of experts and people in these areas disagree with this analogy for fairly deep reasons (I link other resources and discussion here: https://chatgpt.com/share/6ad3aa3d-d110-83ea-a8c5-8653b34743...
This is a really interesting article! While I can’t profess to understand all the math it is a great way to demonstrate it.
I recall awhile back hearing someone ask whether intelligence was actually the same or similar to “compression” or storing all of this knowledge in weights - IE the ability to create something from reason or weights or relationships / connections. This article makes it even more interesting!
For anyone interested, here is a recent 3Blue1Brown series on "Compression is Intelligence": https://www.youtube.com/watch?v=l6DKRf-fAAM
(For anyone not familiar with 3B1B, it's a YouTube channel that visualizes intriguing math concepts to explain them intuitively!)
Funny how age shows up in some cases. For me the first thing that came to mind, when starting to read the article, was Ranulph Glanville's 1978: "What is memory that it can remember what it is?" (Progress in cybernetics and systems research - vol 4)
Not sure if this was a joke or a timeline glitch, but the year was 1976 and the volume was 7 ;)
> Glanville, R. (1976). What is memory, that it can remember what it is? In R. Trappl et al. (Eds.), Recent Progress in Cybernetics & Systems Research, Volume 7 (Proceedings 3rd European Meeting on Cybernetics and Systems Research). Washington, DC: Hemisphere Press.
See https://ranulphglanville.org.za/publications/
Given the down votes I suspect your joke flew over most heads.
"interference-to-signal ratio", "capacity", ... These terms, and their context, are things a communications/electrical engineer would understand. I encourage comms engineers to start learning how AI works under the hood. Chances are, a lot of what you know about maximising channel capacity and throughput is applicable to AI (can be seen as a channel with memory?).
Definitely, not so surprising considering prediction and compression are two sides of the same coin
EE here. Indeed I asked Gemini Argon to translate this article in to DSP terms and it helped.
The animated bird, the kea is absolutely fabulous. I appreciate the effort and detail that went into this.
I was about to return here to say the same, I wonder how many people noticed it!
I didn't, so thanks for pointing out
Great article. My favorite fact about "transformers as a hopfield network" is that transformers are so good at memorizing that they only need one step to find a suitable memory as opposed to multiple steps for a vanilla hopfield network
For anyone here who understands this stuff, help me out because the article dropped me at the end when drawing the connection from memory to LLMs.
For an LLM, the keys are the tokens in your context, and the values are … what exactly?
In an attention layer, the query, key, and value are all computed at inference time using weight matrices that are determined as a result of the training. If X is the token representations entering the attention layer, then each of Q, K, and V would be determined by multiplying X by the corresponding learned weight matrix for queries, keys, and values:
Q = XW_q
K = XW_k
V = XW_v
From there, you can think of K and V like keys and values in a Python dictionary, and Q as the key you pass (e.g. d[Q] in Python to return some value V). The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
> The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
I understand you meant to compare it to dicts, but this is more like (the only?) similarity to outer product memory. Worth calling out since GP is confused about this connection specifically.
2 replies →
To clarify slightly, K and V are used to construct the dictionary, and then Q is the lookup key D[Q] as described (with the lookup being based on similarity rather than exact hash as it would be in an actual dictionary).
See my comments and links below, you are not wrong, IMO this is not actually a great analogy for understanding LLMs (other than as an explanation for why a trained one can so efficiently compress such a large amount of information).
https://news.ycombinator.com/item?id=50131741
Wow!! I wish I had this perspective 13 years ago during those dull courses describing the mechanics of an MLP.
So refreshing to see actual human dialog at the start of an AI post/paper/explainer, instead of having that visceral "I'm just reading AI slop" feeling. Even if it's totally unrelated, the little bit about playing the Dune board game made me much more engaged and willing to listen to what the rest of the content had to say.
See also:
NNCP: Lossless Data Compression with Neural Networks (2019)
https://bellard.org/nncp/
Goodness, someone posts a great article which IMO accessible math, and people get upset.
If you think that is crazy, ohh boy, just wait til you see all the other papers underlying LLMs, AI, and optimization in general.
I appreciate this post because starting from linear algebra is the correct way to form an intuition of the higher dimensional semantic spaces we're dealing with.
Yeah, the notation and math concepts also all just first-year calculus (or analysis, more accurately) and stats (and the latter being only E for expectation). Literally just summation notation, blackboard bold R for the real numbers, double bars for norm / distance, and then max/min with subscript for the quantity being minimized.
Embarrassing to complain about notation so simple, and, frankly, if you can't understand math concepts and notation this basic, you simply can't meaningfully understand modern neural networks: without the math, all you are doing is substituting loose analogies and metaphor for understanding. This is clearly an article for people that want more than that.
This perspective is great.
This sparks a thought that I have not seen revisited in the LLM age - memristors [1].
[1] https://en.wikipedia.org/wiki/Memristor
Who? Me?
[dead]
[dead]
This article does two things which make it unhelpful for me:
1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters?
2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix?
To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings:
> the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning
What made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset.
So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.
> This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters
At the risk of stating the obvious, this is a forum where people can post arbitrary links to other sites, some of which might have audiences that might not be "the exact set of people who read Hacker News". I'm not sure you'd be able to make a compelling argument that alexwang.ai is aimed at software developers and no one else.
Yes, but you need to consider your audience when posting links to this site. For example, if I posted to HN a link to a really good cookie recipe, it wouldn't be a good fit for this site. The author did nothing wrong, but that doesn't make it a good submission to HN.
This article is a bit more of a grey area. It is on a subject that is deeply relevant to this site and I'm sure some HN readers will get value from it. My point is that an even better submission to HN would have been an article on the same topic but written for programmers rather than mathematicians.
23 replies →
FWIW, I found the idea of interference-to-signal ratio as a way to think about memory capacity to be intuitive and was really interested in getting a better sense of how models increase capacity, until it hit the first math notation formula. My disconnect was based on the approachable tone of the intro, from which I assumed it would be a more generally accessible article.
To be clear, I have no problem with math people conveying math stuff with math notation but it seemed like the type of article which, even if it will get into notation, first at least explains "the main idea of it" without notation, then expands with notation as necessary when fleshing details or implementation.
Wait, you're saying that mathematics isn't useful to software engineers?
You should give it a go.
That is not an accurate read of my post. My post was about two things:
I'm not saying they should remove any of the content, just put it in a form more accessible to us. For example, I can read and mostly understand a news article in French, but it will take a lot of time and effort. If it was written in English, it would be easy for me to read. The two news articles have exactly the same content, one is just written in a form easier for me to understand.
Well putting this article in math notation without defining any of the data structures behind it, that's not like putting the article in French, it's like putting the article in Japanese. I can't understand any of it. Some Japanese speakers on this site are saying "I don't understand the problem, just learn Japanese." which is pretty much the least constructive comment they could make.
1 reply →
I guess you're not the target audience for this article then.
This is pretty approachable maths, and explain really well. I've seen far worse in programming language theory papers, which would qualify as "software development" in my mind. There are plenty of users of this forum that would find this paper useful and interesting, with sufficient math background to read it to boot.
Those are valid bits of feedback to make an articel more accessible, but it's totally fine and apprpriate as it is
What made it not useful to you likely makes it much more useful for me.
Fortunately, LLMs can work wonders in adapting content to be more accessible to a wide variety of audiences. Just give it a link and feed your exact post to it. It will be able to translate it into pseudocode and provide examples.
The author has a link to the actual code behind the blog post. https://github.com/KeAWang/associative-memory-blog/blob/main...
Well, you could tell your AI of choice to read the article and explain the math. I did that with ChatGPT and it helpfully translated a lot of it into Python for me.
Upvoting you for this is quite fair. Mathematicians do seem to suffer from the Feldspar Problem:
https://xkcd.com/2501/
>It's easy to forget that the average person probably only knows the formulas for Olivine and one or two Feldspars
That being said the llm is indeed the universal translator — the babelfish supreme — and is excellent at translating whatever you want into pseudocode in your language of choice.
Send the link to Claude or your favorite alternative, and cut-and-paste your comment into the box, and you'll get exactly what you want.
[dead]