Comment by impendia
1 hour ago
Math professor here.
When I go to research lectures, I sometimes hear that in response to audience questions, although not especially consistently. Some speakers do this more than others, I don't think anyone does it all the time.
It was so long ago that the specifics have faded, but I remember I was coached to use a variety of positive responses. "That's a great question," yes, but also things like "I'm glad you brought that up," and "I was hoping someone would ask about that!" It wasn't my cup of tea, too artificial, but the advice was contemplated.
The next question (which is a great one, from what I understand) is: Why do LLMs use these phrases so much if humans rarely use them in written form? Maybe a fair portion of training data comes from lecture transcripts, where such responses are common when responding to direct questions? And/or system prompts are just instructed to be like that?
> Why do LLMs use these phrases so much if humans rarely use them in written form?
As far as I understand, it's due to RLHF. The reviewers the AI companies use don't necessarily know what kind of question is a good one, so when the LLM answers "That's a good question!", they tend to rate the answer higher because they like being flattered. Proxy models that are themselves trained on RLHF inherit this pattern. Similar effects contribute to sycophancy.[1]
[1] https://arxiv.org/abs/2310.13548