Comment by molf
15 hours ago
Why would you tell people that the correct order is to build foundational knowledge before exploring a subject? For some (many?) people, a 'proper' understanding develops _after_ the exploration.
15 hours ago
Why would you tell people that the correct order is to build foundational knowledge before exploring a subject? For some (many?) people, a 'proper' understanding develops _after_ the exploration.
> Why would you tell people that the correct order
Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?
> For some (many?) people, a 'proper' understanding develops _after_ the exploration.
I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".
If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.
> unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring"
That's what I would call exploring.
I too started exploring programming as a teen by cargo culting. Fooling around and getting results is what made it fun. Understanding came later.
> That's what I would call exploring.
Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing.
> Fooling around and getting results is what made it fun. Understanding came later.
It is not how LLMs are "built from scratch".
2 replies →