Comment by BoorishBears
2 days ago
The largest model I've post-trained in the last 2 years of working on this problem was Kimi 2.5 at 1T parameters.
The simplest way I'd put it is, teaching a model to write coherently (follow rules, patterns, etc.) is easy enough: just use teacher forcing. Teaching a model to write creatively is easy enough: just use RL and punish it for not being creative.
Teaching a model to write well and creatively takes learning two partially opposing objectives that spike the learning requirements in ways that smaller models really struggle with.
> is easy enough: just use RL and punish it for not being creative.
How are you scoring creativity in an unsupervised manner? That seems anything but easy.
Did you try reading the whole comment?
Once creativity is being measured in isolation, getting multiple responses from the model is enough to measure creativity a ton of different ways: wordfreq to identify overused phrases, getting multiple responses for the same prompt and promoting the least similar as preferred for policy optimization, etc.
But that's of limited use for stuff like getting diverse names and such. You want creativity and coherency, and if you just punish the model for using an overused phrase, the first thing it does is strongly learn a new overused phrase (or gibberish).
(Also I don't think you mean unsupervised. You probably mean without humans [since LLMs struggle to judge creativity], but that's not what unsupervised means.)
I did read the the full comment and I did in fact mean exactly what I wrote when I used the term "unsupervised". I think the condescension does nothing but get in the way. Try extending the benefit of the doubt.
> enough to measure creativity a ton of different ways ...
The things you listed seem more like temperature than creativity to me. At this point it occurs to me that this is likely yet another case of highly misleading technical jargon. Suffice to say that truly creative writing requires something entirely different than unusual sentence structure - in fact it doesn't require unusual phrasing at all.
Re unsupervised, it seems the misunderstanding here follows naturally from the previous difference in word meaning. Hopefully you see the difficulty of scoring long form answers for the creativity of the underlying ideas, as well as the impossibility of using a labeled dataset to train on such a criteria.
2 replies →