Comment by derefr

> The entropy of ChatGPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It is, ultimately, predicting what will manipulate the imaginary human rater into giving it a high reward.

This isn't strictly true. It is still predicting "most likely tokens"! It's just predicting the "most likely tokens" generated in a specific step in a conversation game; where that step was, in the training dataset, taken by an agent tuned to maximize reward. For that conversation step, the model is trying to predict what such an agent would say, as that is what should come next in the conversation.

I know this sounds like semantics/splitting hairs, but it has real implications for what RLHF/instruction-following models will do when not bound to what one might call their "Environment of Evolutionary Adaptedness."

If you unshackle any instruction-following model from the logit bias pass that prevents it from generating end-of-conversation-step tokens/sequences, then it will almost always finish inferring the "AI agent says" conversation step, and move on to inferring the following "human says" conversation step. (Even older instruction-following models that were trained only on single-shot prompt/response pairs rather than multi-turn conversations, will still do this if they are allowed to proceed past the End-of-Sequence token, due to how training data is packed into the context in most training frameworks.)

And when it does move onto predicting the "human says" conversation step, it won't be optimizing for reward (i.e. it won't be trying to come up with an ideal thing for the human say to "set up" a perfect response to earn it maximum good-boy points); rather, it will just be predicting what a human would say, just as its ancestor text-completion base-model would.

(This would even happen with ChatGPT and other high-level chat-API agents. However, such chat-API agents are stuck talking to you through a business layer that expects to interact with the model through a certain trained-in ABI; so turning off the logit bias — if that was a knob they let you turn — would just cause the business layer to throw exceptions due to malformed JSON / state-machine sequence errors. If you could interact with those same models through lower-level text-completion APIs, you'd see this result.)

For similar reasons, these instruction-following models always expect a "human says" step to come first in the conversation message stream; so you can also (again, through a text-completion API) just leave the "human says" conversation step open/unfinished, and the model will happily infer what "the rest" of the human's prompt should be, without any sign of instruction-following.

In other words, the model still knows how to be a fully-general, high-entropy(!) text-completion model. It just also knows how to play a specific word game of "ape the way an agent trained to do X responds to prompts" — where playing that game involves rules that lower the entropy ceiling.

This is exactly the same as how image models can be prompted to draw in the style of a specific artist. To an LLM, the RLHF agent it has been fed a training corpus of, is a specific artist it's learned to ape the style of, when and only when it thinks that such a style should apply to some sub-sequence of the output.