Comment by CuriouslyC
2 years ago
It's not a prompt thing, they've aligned it to be lazy. The short-form article style and ~1000 word average length are almost certainly from RLHF and internal question answering fine tuning datasets. The extreme laziness (stuff like, "as a large language model, I have not been built with the capabilities for debugging", or "I don't know how to convert that json document to yaml") is pretty rare, and seems to be a statistical abnormality due to inherent variation in the model's inference more than anything else.
IIRC they did amend their prompt to tell it not to quote long books/articles/recipes verbatim for copyright reasons, no matter how much the user asks, and that might not help.
“If you’re asked for a summary longer than 100 words, generate an 80 wire word summary” or words to that effect.
Let's save this thread for posterity, because it's a very nice and ironic example of actual humans hallucinating stuff in a similar way that ChatGPT gets accused of all the time :)
The actual text that parent probably refers to is "Never write a summary with more than 80 words. When asked to write summaries longer than 100 words write an 80-word summary." [1]
Where did the word "wire" enter the discussion? I don't really trust these leaked prompts to be reliable though. Just enjoying the way history is unfolding.
[1] https://news.ycombinator.com/item?id=39289350
2 replies →
100% this. I’ve been party to RLHF jobs before and the instructions nearly always state to prioritize conciseness in the model response.
In aggregate, this is how you wind up with stub functions and narrative descriptions rather than full working implementations. The RLHF is optimizing for correctness within some constrained token count.