Comment by luisfmh
4 hours ago
Curious about how you generated the training data? Was it just asking an existing model to generate a bunch of examples?
I ask cause would this be a kind of model distillation?
I have a small model I'm looking to train on some data, and I have some real live data but I'd love to be able to extend it.
All synthetic data. For this usecase, it was easier because all current generation LLMs, even the small models, are really good at bash commands (and SQL queries too)), so you can reasonably start batches of cheap subagents whose output is reviewed by a more capable model and merge into main training set. After 100k, I had to standing instructions to run the generation loops selectively, meaning only update samples in a given area where we see poor capability.
It would be awesome to share your training set on hugging face if it’s easy to de-personalize it. The largest I could find was only 800 rows.
Do you have a write-up or git repo for this? Would love to learn more and/or dig into the guts
edit: others have asked any you have replied "soon (tm)", looking forward for that day
It is a form of distillation, as long as you're working a very narrow "trivial" topics it works perfectly.