Comment by nialv7
4 hours ago
The funny thing about LLM is, you can build one from scratch and yet you still won't understand how it works. You would understand what kind of matrix multiplications the neural network performs (in fact that's not that hard. An OS is orders of magnitudes more complex), but you would still have no idea why it does what it does.
This is a great point. The complexity sources are entirely different. An OS is complex because it’s solving a difficult problem and we’ve seen decades of evolution both in knowledge and-how and, at the frontier, in performance standards.
The abilities of LLMs are emergent. You can experiment with LLMs and know as much about their observable behavior as the experts. But there’s no way to “crack open” an LLM and see precisely where each skill or tendency lives; as far as we currently understand, it’s all mashed together.
> there’s no way to “crack open” an LLM and see precisely where each skill or tendency lives
Mechanistic Interpretability has entered the chat.
For a classic example, see https://www.anthropic.com/research/tracing-thoughts-language...
The spirit of your point stands, though. This kind of research is interesting to read about, but it's very hard, more like neuroscience or biology than computer science ("LLMs are grown, not made"). You're dealing with a lot of extremely _messy_ complexity, for which organic life is really the only good point of comparison. Most of us here are't really equipped for that kind of work; it's not at all like, say, reverse-engineering a piece of software written by humans. And of course the only people who can do it on frontier models from Anthropic and OpenAI are people within the labs themselves. (But I'm optimistic we'll see more of this work on open weights models...)
Learning how to build emergent systems is also a skill kids should learn these days. The closest I got was coding up game of life for CSE 142 (intro programming).
If stochastic gradient descent isn't taught in whatever CS Theory 101 is now, it really should be these days.
Yah, "building" it is not sufficient. But a lot of times when I build I want to know the why. "Why does gradient Descent have some clever tricks that easily translate to matrix math"?
Lot's of neat stuff to learn.