Comment by aabhay
3 days ago
I think the questionable term is not “syntactically likely” but “merely”. Syntactical likeness is a vast solution space that encompasses the work of a terrible developer and a genius developer. In fact this solution space is a gap wide enough to encompass all coding knowledge and expertise.
"Combinations of the training data" is also hiding a pretty common and serious misconception about modern generative AI: this seems to come from a mental model that LLMs and image generators are merely producing a collage of sorts, especially that this is some way linked to specific parts of the training data as opposed to a generalized result of building up an understanding of how to produce a sentence, piece of code, or plausible image.
That is how they feel when using them for code. It feels like they generate templates filled in with the details of a specific case.