Comment by mgraczyk
4 years ago
I'm disagreeing with the language you are using to characterize models. "copying from memory" implies that there is something being copied, and a memory that you are copying it from. I am pointing out that LLMs do not do this. It's not how they work.
If you polled 1M random English speakers randomly and asked them whether or not a system that "just sort of repeating and copying from memory" could produce completely novel answers in response to completely novel questions, I suspect that the overwhelming majority would respond by saying no.
Similarly if you asked 1000 people working on LLMs whether they work by "copying from memory", I suspect nearly all would say no. It would be accurate to say they are "generating text via a probabilistic model of language, which is encoded in the weights of a neural network", but there really is just no sense in which the models are "copying" anything.
That being said, these models do "copy" some text in the sense that they can reconstruct some strings from their training input. For example every LLM I have played with can recite the first few paragraphs of A Tale of Two Cities verbatim. But that's a capability they have _in spite of_ their actual design, not because of it.
> I'm disagreeing with the language you are using to characterize models. "copying from memory" implies that there is something being copied, and a memory that you are copying it from. I am pointing out that LLMs do not do this. It's not how they work.
Then we're arguing about the semantics of the word "copy." That is not an interesting argument when you know exactly what I mean and can express it clearly.
edit: If it helps, either substitute your description in whenever I say 'pretty much copy' or change the word "copy" to whatever word you want to use. But even though I can't reproduce the opening paragraph to A Tale of Two Cities verbatim, I can certainly write something that is "copying" it without doing that, and anyone who was familiar with the book and read my paragraph would agree with me.
It is semantics, but that was your whole point no?
> That's because they're not modelling anything
If we agree on "how LLMs work", then how can you claim that they aren't modeling anything? They are modeling language, and while it's unlikely current paradigms will be proving new mathematical truths, it's completely plausible to me that bigger models will be able to handle simple math word problems like those in the article, precisely because LLMs can model the "Alice", "Apple", and "Bob" entities.
I disagree that they are modeling language.* I think that not only bigger models but same-sized or much smaller models will be able to handle arbitrarily complicated word problems if they're eventually supplemented with some explicit model-building process.
-----
* ...and that would be a completely semantic argument to have. I don't care whether it's called modeling, other than the fact that when I'm talking about modeling, I'm not talking about language probability, I'm talking about categories. But discussing what current AI is (a language model, copying?) is a waste of time, because I absolutely agree with your description of how it works, so we're talking about exactly the same thing.
The memory is the model. Having high probability to a particular area in latent space is copying it from the memory of the training data.
You're a massive pedant, and this style of comment is a large reason HN comment sections are a pain to read.
What is the difference between this and describing a human brain the same way? The brain is the model, you are "just" copying things from the memory of your brain to words that you speak or write?
I don't think it's pedantic to say that an argument is wrong because it's making an incorrect claim. The claim here is that there is something different or missing between a true "model" and LLMs, and that missing thing has something to do with "copying". But that's not true, the missing thing is the complexity of the table, or the size of the table. The fact that it's copying in some incredibly abstract sense doesn't matter.
i think humans are just copying every time we give an answer without much thought. we're either regurgitating something we previously thought/solved or something we heard somewhere.
when you have to put your head down and actually think for awhile, then maybe you're doing something new or at least not within your brain-training data. I don't think AI can do this yet. it can only copy pieces of its training data out to look like something new, but it isn't really. like when i was describing a video game idea i had to a friend and he called me out on just stealing bits of other games and mashing them together. he was right. it wasn't original. and this is all the AIs can do right now.