Comment by dan_sbl
2 months ago
I took one of the examples ("Leveraging sqlpipe's robust...") and just prefixed it with this:
> Rewrite this using ASD-STE100 simplified technical English:
And you get a good-enough result, it seems? Maybe add another sentence or two for guidance, but what's with needing these giant skills, when ASD-STE100 was surely in the training material?
The training material is used to "develop intelligence" not as reference material.
As an analogy, students learn a bunch of things during school, become more intelligent, but do not retain explicit knowledge like all of the details on the periodic table, or all of the exact formulas for things.
I certainly get what makes you feel that the model should have been wise enough to utilize the knowledge in its pretraining, but it seems like the pretraining data gets pretty muddled in later stages.
The pretraining data isn't getting muddled, the instruction just gets in the lossy middle, and there's less for attention to grab on to. If you repeated the prompt with a hook every turn, it'd stay quite sharp.