Comment by GodelNumbering
4 hours ago
This is the golden age of model training. Some days ago, I decided I wanted a local CPU only model that can perform exceptionally well for English to Bash translation (to avoid the googling for command syntax). I got a bunch of subagents to generate large amount of training data (140k+ samples), got the Qwen 3 0.6B base model, pointed Astra at it, and off to the races. It trained for 2 days (on and off) and I got a surprisingly good model for my task! The total active time I spent was a few hours. And it is still improving, what a time to be alive!
Curious about how you generated the training data? Was it just asking an existing model to generate a bunch of examples?
I ask cause would this be a kind of model distillation?
I have a small model I'm looking to train on some data, and I have some real live data but I'd love to be able to extend it.
All synthetic data. For this usecase, it was easier because all current generation LLMs, even the small models, are really good at bash commands (and SQL queries too)), so you can reasonably start batches of cheap subagents whose output is reviewed by a more capable model and merge into main training set. After 100k, I had to standing instructions to run the generation loops selectively, meaning only update samples in a given area where we see poor capability.
It would be awesome to share your training set on hugging face if it’s easy to de-personalize it. The largest I could find was only 800 rows.
Do you have a write-up or git repo for this? Would love to learn more and/or dig into the guts
edit: others have asked any you have replied "soon (tm)", looking forward for that day
It is a form of distillation, as long as you're working a very narrow "trivial" topics it works perfectly.
I don't understand. If you have a model that can do bash examples already (your subagents), then why would you need to train a model?
Or are the subagents generating your training data using a closed/paid model?
A very small, highly specialized model can use negligible resources (CPU, energy) to accomplish the same task.
For everyday work that happens frequently it's better to have a tiny specialized model instead of making billable API calls or turning your laptop into an 80W space heater for 20 seconds to run a general purpose model.
The large models can be used to generate synthetic training data. Tell them to make up 100,000 tasks paired with the resulting output as a 1-time cost. Then use that to train a small model.
Think of it as distillation, but focused on a specific task.
Given that they're just using it to avoid the googling for bash command syntax, I'm not sure they'll save in the end against the 140k training examples they generated.
4 replies →
This is so cool - I'm aware of this in a vague way. Can you write a little tutorial or give some good links. I want this to be the next new things I do :)
1 reply →
The models he is using to generate training data are presumably commercial models. He is distilling their bash knowledge into a much smaller model he can run locally fast and cheap.
Did your Astra do any RL or just SFT? did it make up any benchmark to ensure the fine-tuning was a success?
If it’s one of thing that you want just for English to bash shell commands, I will create AST, it is deterministic, exceptionally fast, no tokens so no need to fine tune existing model, please let me know your thoughts.
Sorry for the aside, but I noticed half the usecase of AI is fixing the awful DX.
I appreciate the aside. Interesting observation
I'm literally working on context/harness engineering right now (a set of opencode plugins)
Aside on the aside, I welcome this new era of really personal software. Not Ai's being sycophants, rather being able to easily and quickly change, adapt, or extend software I am not familiar with.
That's a really impressive result. There are all kinds of small tasks like this I use an LLM for, but theoretically if you broke all the sub-use cases into local-only models, and had something lightweight that routed to the right model, you could have faster and cheaper workflows. E.g. something trained on the linux man pages for common commands, since it's usually quicker to ask an LLM for a specific command with flags than to consult the man pages.
> That's a really impressive result.
we dont know what the result is and how its impressive.
what hardware are you using to train?
I didn't have a local GPU, so I asked it to go out and find hardware. It found a google TPU v6e which seemed reasonably priced. I gave it my google api key. I told it to use TPU only when training and bring it down afterwards. That's about it.
> I gave it my google api key
This is the part where the narrator looks at the camera and says "Don't try this at home, kids!"
6 replies →
What kind of observability did you have over this process? I’m interested in how my peers are operating these efforts.
2 replies →
> I told it to use TPU only when training and bring it down afterwards.
I wouldn't put my house on it. Brave.
Neat!
Seriously, I'm using a Qwen 3.8 27B on the homelab, distilled from supposed Fable traces. Regardless, the difference is notable, less thinking, better output. Distilled / heavy quant is better than the original (imv)
https://huggingface.co/vwdubb/Qwen3.8-27B-Fable-Distill-NVFP...
side quest, are fable distillations only wrong when it's another country?
i also need more info!
I am thinking about opensourcing everything, although this is not my main domain or my main startup, so the overhead of huggingface etc seems a bit unnecessary
Edit: will do as soon as possible
just ask the agent to write it up if you don't have time to do a write-up yourself
+1, would like to see. Even if it's not fully "ready for consumption", it's probably enough to reproduce the results.
Please do! Small, specialized models need more love and the time you spent would be a gift!
Would also love to read a write-up about this!
> what a time to be alive!
It's good to hear you're enjoying yourself, but I suggest retiring that expression. It's really beginning to grate.
Golden age before the age that ends humanity. Not talking about any "rogue AI", just the known statistical models of what is coming due to climate change.
Do those statistical models account for declining birth rates or are they based on prior population growth projections?