Comment by faxmeyourcode
7 hours ago
This weekend I just cracked into nanoGPT (https://github.com/karpathy/nanoGPT), an older but fabulous learning exercise where you build and train a crappy shakespeare GPT with ~0.8M parameters on a cpu. Results are about what you'd expect from that, they suck, but you can start to feel the magic, especially if you're not a deep learning professional and you just want to poke around and hack on it.
I started writing up a blog post on my weekend with nanoGPT but it's not done yet... Would have been great to link to here lol oh well
It's a useful exercise. A lot of the good ML work is first validated at small scale.
And this new example goes even further - adds instruction following and tool use SFT, as well as RLVR. Makes for a more useful baseline.
the shakespeare code tuned a little with different training data does a good job of generating Magic The Gathering commander decks
would love more details on this. this is exactly the type of project I'd like to dabble in to get more up to speed.
FWIW, there was a pretty popular post on HN around generating MTG cards using AI a couple years back but I believe that their approach was a fine-tune on an existing LLM.
https://news.ycombinator.com/item?id=37427854
I like the idea of specific-purpose toy models. How did you tune the code and what dataset you used?