Comment by danielmarkbruce
1 day ago
Get Sebastian's book and build an LLM from scratch. The problem with most textbooks (on any subject, it's not unique to ML/AI) is they usually try to build the theoretical foundations first. In many cases things are better taught top down.
I'm OK going through the learning itself (will definitely check out the book though, since it's so well liked!). The question is more, what do you do with that once the book or course is done?
For a junior coder, there's usually a path to recommend, joining some open source project or doing some home scripting, etc. Something where you can be useful even if it's not your full time job.
I'm wondering whether there's some similar hobbyist path available here, or whether there's too much of a jump between the educational/pop sci stuff and the professional realm. If it's the second, that's also fine, I just want to know if I'm missing something.
One practical-ish thing you could reasonably do with this (beyond just better understanding how models work so you can make better choices/tradeoffs), is post-train some of the small-to-medium models, fine-tune them on your own data or priorities.
You don't need to change many weights, relatively, to add capabilities or modify behavior. Look up LoRas for instance.
I'm of the opinion that post-training existing models is an area with a lot of benefit and fun in the near future (it's already being done here and there of course).
I doubt you _really_ have to know everything in this course to know how to do that, but I'd want this as background personally for it.
You included machine learning in with LLMs in your root post. Try training a supervised model for some task, like an image classifier or a detector. LLMs have not replaced convolutional models, and even transformers can be used for non-NLP tasks. The idea this is somehow not professional is not true. Despite what people would have you believe, ResNet50 fine-tuned on a good dataset is still competent. In many environments there isn’t enough compute for bigger models, and large language models aren’t always the best choice.
You also don’t have to go straight into LLMs. You can learn most of the training mechanics via classic deep learning. It isn’t much different with LLMs except the training data and losses are specific and the models are much larger.
The comment below/above is right. Pre-training isn't the path (but you really do need to learn how it's done). Learn how it works, learn the architecture, then take models and do post training. Fine tuning and RL on them. There was a fun/clever/useless project recently posted here about building a terminal emulating model thing, as an example.
In many cases if you follow a recipe without understanding you're no better off in the end. In cooking at least you can have a cake and eat it too.
i am actually looking for a book that follows like sebastian books but without any hands on coding . i am hoping to read in bed.
There's a note in Sebastian Raschka's LLMs-from-scratch github repository - "Recommendations for Getting the Most Out of a Technical Book".
He recommends reading offline and code-free first (although chapter-by-chapter).
https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a...
Ask and... ye shall receive! Check out `Hands-On Large Language Models: Language Understanding and Generation`.
...I wanna mention this book as well, although it doesn't fit the criteria: https://www.thelittlelearner.com[
You don't need to code to follow it, just read along
I read it in a cabin on a lake, far from any internet access, and without a computer. I still found it quite useful.