Comment by kalms
14 hours ago
Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful.
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute
I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
https://thinkingmachines.ai/tinker/ https://github.com/thinking-machines-lab/tinker-cookbook
(As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering").
TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this.
One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later.
There are a lot of other high level abstractions here to look at. Prime Intellect has one.
The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.