Comment by bcooke
1 day ago
Very well said.
And when you say it like that, I have to wonder how much of this is a natural consequence of RHLF on such a grand scale, when you have millions of people pretty much much skimming chat responses or operating outside their depth and giving unqualified feedback to the models.
Seems like a lot of people may be reinforcing what sounds smart over what is smart.
Also as an aside: funny how much the LLMs continue to mirror the human communication they’re trained on
I believe we are several generations past peak-RLHF at this point. Now it's much more RLVR (Reinforcement Learning with Verifiable Rewards), with a goal/evaluator loop.
Which, conveniently, fits neatly into the benchmaxxing arms race/agentic coding market fit, since you can basically train "directly" on a specific problem space for a benchmark/agentic goal (fudged sufficiently to avoid excess overfitting on public problems/bechmaxxing accusations if real world performance falls short).
The language evolution could be explained by reliance on ever increasing layers of a model judging a model, using a model developed eval, based on synthetic data from a model, etc. And by the time a human evaluator sees it both A/B choices already converged into weird Claude pseudo English as that was baked in much earlier in training.
This, 100%. I don’t think the industry knows how to scale LLMs’ general intelligence much further. The training paradigm is about maximizing very specific behaviors / very specific tasks, but doing lots and lots of them. Which can create the illusion of general intelligence if your tasks are similar to the ones the models were fitted for.
If you have watched The Substance, the transformation feels a bit like when things start falling apart in that one.
I tend to agree. We will see this demonstrated in novel research done by agents, or, more meta-cognitively research direction guidance.-
The model judging a model theory is 100% spot on.-
> or operating outside their depth and giving unqualified feedback to the models
I wonder if the labs are sufficiently prepared to filter this kind of stuff out. I see a lot of non-developers asking development things of Claude, getting confused when they're in over their depth, and getting upset that they don't understand what the model is providing them, giving it bad feedback, and subsequently making the AI worse for the rest of us who know how to use the tool.