Comment by kevmo314
16 hours ago
That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.
16 hours ago
That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.
You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market.
Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to more resources. It has very good prospects if you do manage to push through. But it's a hard and risky path, and you will not be able to get any interesting results for the longest time.
For a young middle-class student, it just doesn't make much sense. You can do much more impressive and impactful things with your time without getting into that black hole.
I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex. But they can't do much with it without capital or access.
Good engineering has never been a bottleneck in this field, it's been all about having access to capital and taking smart but dangerous risks burning it on compute, without much idea of how long you need to keep burning for. There's still no end in sight, some are still managing to convince investors and keep burning, and we are seeing progress, but the business case is still unclear. If you want to get in that game, go ahead, but it's not something I would advice the average young engineer.
> I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex.
I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes.
On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….
The world needs pragmatic folks who work at a higher abstraction and make LLMs useful, AND also folks who think why not “this other way”? And build newer ways to do fundamental things.
Given the usefulness of current LLMs, I would certainly encourage anybody to try and build their own LLMs, and see what they come up with…
Heck if they build a rack full of old laptops and run something with it that could be done “better” with modern servers, I’d still appreciate the learning running things on those little machines bring.
Well, that's not how it works. You don't just put some old laptops into a rack.
Maybe with a decent consumer GPU like a 4090, you could do experiments like distilling and fine tuning a small image model for edge deployment for specific tasks.
Even there, many use cases might require renting compute for $10/hour and investing a few hundred.
A LLM from scratch? Forget it. You can do theoretical experiments, but not build anything remotely useful with that kind of budget.
If you're talented enough to come up with revolutionary methods, maybe an university or AI lab would be the place to be.
1 reply →
To me the bottleneck is not even the compute, which is an issue for sure, but the data. All these large companies got their hands into petabytes of data, a lot of which of illegally acquired, but now they are large enough to pay the fines.
> On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….
10000%.
1 reply →
Agreed. It's hard to learn unless you have access to quite high end hardware, and even paying by the hour is expensive. There's a low ceiling on what you can learn without doing training runs.
You can however learn everything you need to know to get on the career ladder as a software engineer on a regular home PC.
While the topic here is narrow, the concept is broader.
Do you take the first step or rule it out because you don’t yet see the complete picture.
As a teenager I never hesitated to try things out. As a young adult I wanted the whole picture. Now I’m back to playing / trying things out. I kinda wish I’d not given it up. PG being a bit older and reminiscing - I bet he’s in that bucket too, whereas someone trying to establish themselves professionally probably (aka me early 20s) wants to see the path.
> . But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large"
The “large” qualifier dates back to pre-transformer language models, where even training a multi-million model was hard due to how poorly it scaled. GPT-2 was a large language model, despite being only 124 millions parameters.
Due to how much high quality data is readily available, anyone can now train a sub-billion (L?)LM on commodity hardware.
And I'm personally convinced that pretty much any enterprise use-case of an LLM (except coding) is better served by a fine-tuned small (<2B) model that is trained specifically on the task, rather than a generalist frontier model, so learning the engineering around fine-tuning is a key skill that companies will realize they need sooner than later.
Why? Oersted is correct, for any size class you can find an LLM that is free and well trained at this point. They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also. And real world fine tuning is mostly about data gathering and cleaning. The actual adapter training is automated and put behind simple APIs.
I saw my first language model in action in 2014, I was writing blog posts about them back in 2016. In recent years, like many of us, I spent some time learning ML frameworks to see if it'd be a fun career pivot.
But:
1. It doesn't seem especially creative. All the frontier labs have nearly identical model personalities, capabilities and even app designs. We're not seeing them differentiate from each other, implying that the design space might not be that large. In which case the opinions and unique approaches of specific engineers aren't that important, they are interchangeable at the right level of skill, and what to do next is usually obvious to everyone.
2. It doesn't seem like a big job market. A lot of ML jobs were wiped out in recent years by the rise of LLMs. Lots of NLP specialists etc were suddenly replaceable with a cheap API. The jobs that remain have compacted into a small number of companies. It's a small community which greatly increases career risk, especially as so many are unprofitable and/or have strong ideological requirements.
3. It's unclear how much demand for better models there actually is. Do we actually need smarter models? In robotics clearly yes and robotics is interesting and high potential, but for pure LLMs/image models, most users are already incapable of setting tasks that stress the best models and are happy with the cheaper smaller ones.
Using the models on the other hand is a very large design space, and has a lot of scope for creativity. I see use cases for AI everywhere, but most companies seem to stop at putting a chatbot on their website or asking Copilot to rewrite an email before they send it. A lot of companies have hollowed out their IT departments over the past twenty years. It feels like a new golden age of consulting work could be upon us.
I really don't think I'd tell a 17 year old to learn how to train LLMs. Learn how they work and how to use them, sure, absolutely.
7 replies →
But can you _sell_ that? If you can't you can't get a job doing it.
I suspect most who worked at Google did not work on Google infrastructure before getting there.
No, but the skills Google needed (back then) were just normal programming and sysadmin skills scaled up. I had eight interviews that covered Linux sysadmin, programming, debugging, networking, maths and more. If Google hadn't wanted me there'd have been plenty of other companies who needed those skills.
If you look at model training jobs a lot of the work at this point is creating RL gyms (normal programming work), but most people still think the work is all neural architecture research. Doing the former is fine but won't teach you much about how to build LLMs, whatever that means now. Doing the latter is a very hard market to get into: not many jobs and requirements are often like, "you must have published at one of the following conferences". Prior experience is assumed. Most of them seem to treat Google as ML university and source of new recruits. It's understandable given the cost of training runs.
3 replies →
Interesting/capable diffusion models are much smaller than similarly interesting language models. But yes you could always scale things down to learn the fundamentals.
There are plenty of similarly tiny language models in the realm of tts too. Gatekeeping what’s interesting misses the forest for the trees.