Comment by bamboozled
2 years ago
I wonder if there is a hard coded prompt somewhere prompting the model to be "lazy" by default, to save money on inference, or something like this. Maybe not how it works?
When you ask if to write the complete code, it just ignores what it was originally told and does what you want.
It's not a prompt thing, they've aligned it to be lazy. The short-form article style and ~1000 word average length are almost certainly from RLHF and internal question answering fine tuning datasets. The extreme laziness (stuff like, "as a large language model, I have not been built with the capabilities for debugging", or "I don't know how to convert that json document to yaml") is pretty rare, and seems to be a statistical abnormality due to inherent variation in the model's inference more than anything else.
IIRC they did amend their prompt to tell it not to quote long books/articles/recipes verbatim for copyright reasons, no matter how much the user asks, and that might not help.
“If you’re asked for a summary longer than 100 words, generate an 80 wire word summary” or words to that effect.
3 replies →
100% this. I’ve been party to RLHF jobs before and the instructions nearly always state to prioritize conciseness in the model response.
In aggregate, this is how you wind up with stub functions and narrative descriptions rather than full working implementations. The RLHF is optimizing for correctness within some constrained token count.
It's probably just a result of the training data. I bet its not explicitly "trained" to reply with 400 loc for a complete file, but its trained to return a few dozen lines of a single method.