Comment by bevekspldnw
2 days ago
These aren’t raw base models they are the result of a ton of RLHF and various adjustments.
Bitter lesson wildly overstated in this context.
2 days ago
These aren’t raw base models they are the result of a ton of RLHF and various adjustments.
Bitter lesson wildly overstated in this context.
rlhf = reinforcement learning from human feedback
(had to look it up)
I think it's more RLVR (reinforcement learning from verified rewards). The RLHF is just to align models to human preferences, meaning to behave nice.
More accurate to say RLHF aligns models to human preferences, most significantly to be helpful.
What makes you say that
Not sure where I implied they are "raw base models" and not sure what "various adjustments" means here or how "ton of RLHF" contradicts anything. If we look at research for open source models, "adjustments" usually come in the form of efficiency gains which directly contributes to the ability to scale or synthetic data pipelines to increase the dataset and increasing the context window.
More RLHF is in fact scaling.
Yes, but not in the “dump another chunk of all written language in the bucket and stir”-sense which is what bitter lesson became synonymous with.
That may not be the intent of the original article, but over the past few years that’s what the phrase turned into.
GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.
7 replies →
The bitter lesson just means “compute scaling beats hand-tuned architectures in the long run”.
As GP said. More RLHF is in fact the bitter lesson.
Not just RLHF but also RLVR, and isn't that the litter lesson though?
My sense of the Sutton Dwarkesh interview was that he was calling out that he didn't mean just longer datasets, but rather learning through exploration and that's exactly RL.
They just need more contact with reality. That's what RL is right? Contact with narrow subsets of reality.
RLHF is an increasingly small part of training though? From what I understand most of the capability gain is in RLVR
Nah, the last few generations have more RLVR in the data mix. Which is more CPU intensive and very much amenable to the bitter lesson as you can reduce the loss by doing more rollouts in your tool environment.
The scaling with reasoning models is more and more with things like verifiable rewards (coding and math), in line with bitter lesson and also Sutton invented lots of modern RL.