Comment by nikkindev

Author here: Thanks for the explanation. Intuitively it does make sense that anything done during "post-training" (RLHF in our case) to make the model adhere to certain (set of) characteristics would bring the entropy down.

It is indeed alarming that the future 'base' models would start with more flattened logits as the de-facto. I personally believe that once this enshittification is recognised widely (could already be the case, but not recognized) then the training data being more "original" will become more important. And the cycle repeats! Or I wonder if there is a better post-training method that would still withhold the "creativity"?

Thanks for the RLHF explanation in terms of BPE. Definitely easier to grasp the concept this way!