I always thought it could be because volume-wise, most English prose is probably marketing copy and actual clickbait; so when you train on the entire Internet, you get a troll adept at writing ads. Then people ask AdBot2000 to write a novel and are upset it reads like the next iPhone launch site.
Nah, I think this is a common misunderstanding of how LLMs work, where people think that they mimic the pre-training data. Stylistically everything you see is an artifact of post-training, which is from reinforcement learning not from absorbing mass amounts of text. At some point a person or more recently a bot gave a thumbs up to an A/B tested response including em-dashes and claudisms galore.
> Stylistically everything you see is an artifact of post-training,
It is still not exactly clear if it is true or not. Unless we have base "pt" snaphot of Claude we can't say one way or another. I've played a bit with base models of Nemo, Gemma etc and they all had tics, not much different from RLHFed instruct versions.
So question then, why is it so hard to make an ai that doesn’t do these things? And why do Claude and ChatGPT have the same -isms? They’re both doing the same a/b post training with the same decisions?
To me it has a writerly New Yorker vibe to it, as in the magazine which reads as “polished” and probably performs well in RL but is totally exhausting to read in long sessions and completely inappropriate for coding where precision is paramount above all. In writing terms its called purple prose.
You’re more right than you probably realize!
from a few days ago https://news.ycombinator.com/item?id=49388752
I always thought it could be because volume-wise, most English prose is probably marketing copy and actual clickbait; so when you train on the entire Internet, you get a troll adept at writing ads. Then people ask AdBot2000 to write a novel and are upset it reads like the next iPhone launch site.
Nah, I think this is a common misunderstanding of how LLMs work, where people think that they mimic the pre-training data. Stylistically everything you see is an artifact of post-training, which is from reinforcement learning not from absorbing mass amounts of text. At some point a person or more recently a bot gave a thumbs up to an A/B tested response including em-dashes and claudisms galore.
> Stylistically everything you see is an artifact of post-training,
It is still not exactly clear if it is true or not. Unless we have base "pt" snaphot of Claude we can't say one way or another. I've played a bit with base models of Nemo, Gemma etc and they all had tics, not much different from RLHFed instruct versions.
Yes. This completely explains sycophancy at least.
So question then, why is it so hard to make an ai that doesn’t do these things? And why do Claude and ChatGPT have the same -isms? They’re both doing the same a/b post training with the same decisions?
2 replies →
There's layers, some of token selection is fingerprinting https://github.com/google-deepmind/synthid-text
1 reply →
No, there's no reason chatbot behavior would have anything to do with frequency of text in pretraining.
It's more likely that this is from the training data if they're being trained on reams of Internet stuff.
To me it has a writerly New Yorker vibe to it, as in the magazine which reads as “polished” and probably performs well in RL but is totally exhausting to read in long sessions and completely inappropriate for coding where precision is paramount above all. In writing terms its called purple prose.
https://en.wikipedia.org/wiki/Purple_prose
The New Yorker may be pretentious but it's generally not unreadable like Opus.
1 reply →
Isn’t most of the internet slop by now? Self-reinforcing feedback loop.
See: upvotes here
It's not clickbait, it's automated empathy!
/s