Comment by jameshart
4 hours ago
This sort of ‘it’s probably in the dataset’ thinking is a bit of a relic from the ‘stochastic parrot’ era of LLM critique.
Reinforcement learning doesn’t work by training an LLM on a series of inputs like ‘rewrite python in befunge’ paired with a prepared model implementation. It doesn’t matter if an open source implementation was in the dataset that trained the language model. All it got from that was an appreciation for syntax and language idiom; when it is generating code it’s doing so according to the approaches and patterns that it’s been reinforced into following in the context of a coding prompt - regurgitating existing codebases is undesirable so it is a behavior it will have been reinforced against.
People regularly write here that nobody really knows how LLM weights are work really, we only know how to achieve that and run inference. Isn't that true?
I see it this way in layman sense: - you feed the model with data; - you run the query; - you activate a network and it gives you most probable token, refeeds it back and generates next and so on; - nobody knows exactly why and how this next token comes and can tell strictly which input data gives which output data without running the exact query it with 0 temperature.
So yes, it's not an exact copy, but you also can't say that a particular repo is not reflected anyhow in weights therefore producing a more probable tokens based on the data it consumed during training on it which is partially the actual implementation itself.
I didn’t plagiarize! I merely have deep appreciation for the syntax and language idioms of the New York Times.
I don't think they are saying its regurgitation, but rather that the "approaches and patterns" used to build a python interpreter are in the training data because the training data asks the llm to build one.
>All it got from that was an appreciation for syntax and language idiom;
Sounds like it's you who is behind the times. With things like REA we know that LLMs can reverse engineer the behaviour of binaries and basically treat them as a form of specification to rebuild an application.
This means that old python interpreter acts like a specification for what kind of features must be included inside the project.
LLMs are amazing at input to output transformations and that hasn't changed at all.
I think it's you who has to update his mental model away from "stochastic parrot era" accusations.
PS: if you're trying to accuse me of being a "stochastic parroter", that's exactly what I mean.
Taking an input and transforming it to an output is the exact opposite of repeating something.
Well then ‘it’s in the dataset’ doesn’t matter. An agent can just pull down the code as part of its research when it’s tasked with a similar rewrite. That seems far more useful.