Comment by c7b
5 hours ago
Sounds a bit like 'I want to make horses faster, surely I won't need mechanical engineering knowledge'. We don't know everything that we don't know, so it's hard to say what we don't need to know.
5 hours ago
Sounds a bit like 'I want to make horses faster, surely I won't need mechanical engineering knowledge'. We don't know everything that we don't know, so it's hard to say what we don't need to know.
But I am not asking for a faster horse. I am asking for a draft horse instead of a race horse. I know the tasks I have on hand, I know the VRAM and compute budget I have to run them. I am not asking for AGI, I just want something to edit my little text files.
At a more technical level, what do you suggest? Training a small LLM on Python code exclusively? And then one on general CS/algorithms, which you'll also need? I don't think the current transformer architectures would compose as you suggest.
With what I know about how LLMs work now, I guess I am suggesting more specific variants. Qwen3.8 has a 2.4T version and a 27B version. I understand that to mean that they are the same architecture, just one version has a massive training set and the other has a very small subset. So, it seems very possible that variants of 27B could be generated that tune it for specific things by selecting different training data from the large corpus. One model for Python, another for Swift, another for research, another for creative writing, etc.
I think you're right that current architectures don't compose like that - but I feel like that's a result of the focus on MOAR DATA, and a "race for AGI" - if we set those ideas aside, a more composable architecture seems very possible.