Comment by zahlman

Traditionally (he said, referring to prior art in a field that has basically only existed for about 3 years), this sort of "flattery" was understood (he said, referring to random rumours he'd read on the Internet) to make a big difference, since otherwise the LLM might roleplay as something else. Presumably the RLHF training is stronger now.