← Back to context

Comment by mike_hearn

14 hours ago

Why? Oersted is correct, for any size class you can find an LLM that is free and well trained at this point. They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also. And real world fine tuning is mostly about data gathering and cleaning. The actual adapter training is automated and put behind simple APIs.

I saw my first language model in action in 2014, I was writing blog posts about them back in 2016. In recent years, like many of us, I spent some time learning ML frameworks to see if it'd be a fun career pivot.

But:

1. It doesn't seem especially creative. All the frontier labs have nearly identical model personalities, capabilities and even app designs. We're not seeing them differentiate from each other, implying that the design space might not be that large. In which case the opinions and unique approaches of specific engineers aren't that important, they are interchangeable at the right level of skill, and what to do next is usually obvious to everyone.

2. It doesn't seem like a big job market. A lot of ML jobs were wiped out in recent years by the rise of LLMs. Lots of NLP specialists etc were suddenly replaceable with a cheap API. The jobs that remain have compacted into a small number of companies. It's a small community which greatly increases career risk, especially as so many are unprofitable and/or have strong ideological requirements.

3. It's unclear how much demand for better models there actually is. Do we actually need smarter models? In robotics clearly yes and robotics is interesting and high potential, but for pure LLMs/image models, most users are already incapable of setting tasks that stress the best models and are happy with the cheaper smaller ones.

Using the models on the other hand is a very large design space, and has a lot of scope for creativity. I see use cases for AI everywhere, but most companies seem to stop at putting a chatbot on their website or asking Copilot to rewrite an email before they send it. A lot of companies have hollowed out their IT departments over the past twenty years. It feels like a new golden age of consulting work could be upon us.

I really don't think I'd tell a 17 year old to learn how to train LLMs. Learn how they work and how to use them, sure, absolutely.

7 comments

mike_hearn

Reply
inciampati  11 hours ago

> We're not seeing them differentiate from each other, implying that the design space might not be that large.

There is the more likely reason they are not differentiating. They use almost exactly the same class of model. Everything is linear, parallelizable. It's incredible path dependence that's now invisible enough we think it's a natural law. Nature is not linear.

  • godshatter  3 hours ago

    Exactly. They had the capital to gather massive amounts of data and compute and took a path that went well and pushed it to it's max. Are still pushing it. Are there better ways that don't need as much data and compute? Well, that's for the 17-year olds who don't have trillions to spend to figure out. I'm hoping they do so in a way that brings back affordable gaming gpus and memory sticks as a thing again.

stymaar  11 hours ago

> They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also.

Good luck relying on in-context learning for a 600M LLM.

> The actual adapter training is automated and put behind simple APIs.

That's like saying it's worthless to learn infra because you can use serverless instead…

> All the frontier labs have nearly identical model personalities, capabilities and even app designs. We're not seeing them differentiate from each other, implying that the design space might not be that large

The design space for a generalist model isn't large, by definition. But the design space for specialized smaller models is much larger. If you can train a 200M model that, for your use-case, is competitive with a frontier one, then you'll make your company save a lot of money in tokens.

> 2. It doesn't seem like a big job market. A lot of ML jobs were wiped out in recent years by the rise of LLMs. Lots of NLP specialists etc were suddenly replaceable with a cheap API. The jobs that remain have compacted into a small number of companies.

We are in a strange place where a few companies are collectively burning a hundreds of billions a year to sell things a few pennies for the dollar. Of course it's going to be cheap and concentrated. How is it supposed to end though?

> 3. It's unclear how much demand for better models there actually is. Do we actually need smarter models?

That's the thing actually: I don't think we need better models this much, and if we don't need better models we need the cheapest possible model for a given use-case.