Comment by vlovich123
3 hours ago
> The "does not rescue it". No human would write like that.
This is what I don’t understand. Supposedly LLMs are trained on human text. Why do they come up with such unrealistic prose? Is it intentional because the companies want the tells to be obvious?
They’re not just trained on human prose. They’re sent to RLHF, and also their language changes as a result of RL on verifiable rewards.
Getting it to write well is really hard because there’s no real way to verify whether it’s good prose or not. You and I can tell, but we can’t write a verifier that codifies our judgment.
Maybe they’ll find a way to improve this, but for now it’s certainly one of the harder problems to solve for LLMs.
Part of it is that I think they also have poor theory of mind, which I imagine is also a hard thing to train it to do.
Why is it hard? Ask it to write professionally in mid-twentieth century style English, and without resorting to the clickbait style of writing.
In any event, other LLMs may not automatically have the problem, and don't even require such a prompt. This is a Claude problem.
I don’t think you realize how long ago the middle of the 20th century was.
1 reply →
For autoregressive models (practically all hosted ones), it's because of the nature of next-token prediction. LLMs lock themselves into a particular sentence structure ahead of time and have to guess at the rest of the sentence. Samplers have no insight into the LLM's "intent" aside from the probability of each next token, and the LLM has no insight into its previous "intent" that resulted in a given probability in the first place. I don't think this is possible to solve with more training, I think a fundamental architectural shift will be needed, like more research into diffusion language models.
[dead]