Comment by alexgartrell

https://thinkingmachines.ai/tinker/ https://github.com/thinking-machines-lab/tinker-cookbook

(As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering").

TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this.

One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later.

There are a lot of other high level abstractions here to look at. Prime Intellect has one.

The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.