Comment by reacharavindh
15 hours ago
> I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex.
I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes.
On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….
The world needs pragmatic folks who work at a higher abstraction and make LLMs useful, AND also folks who think why not “this other way”? And build newer ways to do fundamental things.
Given the usefulness of current LLMs, I would certainly encourage anybody to try and build their own LLMs, and see what they come up with…
Heck if they build a rack full of old laptops and run something with it that could be done “better” with modern servers, I’d still appreciate the learning running things on those little machines bring.
Well, that's not how it works. You don't just put some old laptops into a rack.
Maybe with a decent consumer GPU like a 4090, you could do experiments like distilling and fine tuning a small image model for edge deployment for specific tasks.
Even there, many use cases might require renting compute for $10/hour and investing a few hundred.
A LLM from scratch? Forget it. You can do theoretical experiments, but not build anything remotely useful with that kind of budget.
If you're talented enough to come up with revolutionary methods, maybe an university or AI lab would be the place to be.
[dead]
To me the bottleneck is not even the compute, which is an issue for sure, but the data. All these large companies got their hands into petabytes of data, a lot of which of illegally acquired, but now they are large enough to pay the fines.
> On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….
10000%.
One of the things to think about when it comes to many kinds of "expensive" technology, is that from so many well-funded ventures, from the capitalists on down almost every decision-maker involved has not spent the majority of their life making every dollar count in some way or another.
Even more so when things are not just expensive by nature, but truly overpriced beyond that point.
>10000%
Once in a while you do get somebody who only spends a dollar and gets more out of it than a seasoned high-roller spending $10000. Most of the time the waste is borne by those who can afford to throw away $10000 more easily than an economizer can afford to lose one dollar, so nobody is crying about it.
With how ridiculously large the language models have gotten though, a 10000x improvement in actual intelligence does seem like it could be lurking unrecognized at a different point on the compass.