Comment by jameshart
3 hours ago
This sort of ‘it’s probably in the dataset’ thinking is a bit of a relic from the ‘stochastic parrot’ era of LLM critique.
Reinforcement learning doesn’t work by training an LLM on a series of inputs like ‘rewrite python in befunge’ paired with a prepared model implementation. It doesn’t matter if an open source implementation was in the dataset that trained the language model. All it got from that was an appreciation for syntax and language idiom; when it is generating code it’s doing so according to the approaches and patterns that it’s been reinforced into following in the context of a coding prompt - regurgitating existing codebases is undesirable so it is a behavior it will have been reinforced against.
I didn’t plagiarize! I merely have deep appreciation for the syntax and language idioms of the New York Times.
I don't think they are saying its regurgitation, but rather that the "approaches and patterns" used to build a python interpreter are in the training data because the training data asks the llm to build one.
>All it got from that was an appreciation for syntax and language idiom;
Sounds like it's you who is behind the times. With things like REA we know that LLMs can reverse engineer the behaviour of binaries and basically treat them as a form of specification to rebuild an application.
This means that old python interpreter acts like a specification for what kind of features must be included inside the project.
LLMs are amazing at input to output transformations and that hasn't changed at all.
I think it's you who has to update his mental model away from "stochastic parrot era" accusations.
PS: if you're trying to accuse me of being a "stochastic parroter", that's exactly what I mean.
Taking an input and transforming it to an output is the exact opposite of repeating something.
Well then ‘it’s in the dataset’ doesn’t matter. An agent can just pull down the code as part of its research when it’s tasked with a similar rewrite. That seems far more useful.