Comment by stymaar
10 hours ago
> They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also.
Good luck relying on in-context learning for a 600M LLM.
> The actual adapter training is automated and put behind simple APIs.
That's like saying it's worthless to learn infra because you can use serverless instead…
> All the frontier labs have nearly identical model personalities, capabilities and even app designs. We're not seeing them differentiate from each other, implying that the design space might not be that large
The design space for a generalist model isn't large, by definition. But the design space for specialized smaller models is much larger. If you can train a 200M model that, for your use-case, is competitive with a frontier one, then you'll make your company save a lot of money in tokens.
> 2. It doesn't seem like a big job market. A lot of ML jobs were wiped out in recent years by the rise of LLMs. Lots of NLP specialists etc were suddenly replaceable with a cheap API. The jobs that remain have compacted into a small number of companies.
We are in a strange place where a few companies are collectively burning a hundreds of billions a year to sell things a few pennies for the dollar. Of course it's going to be cheap and concentrated. How is it supposed to end though?
> 3. It's unclear how much demand for better models there actually is. Do we actually need smarter models?
That's the thing actually: I don't think we need better models this much, and if we don't need better models we need the cheapest possible model for a given use-case.
There is no reason to believe that generalist models need to be transformer (or transformer hybrids) it's just that people are risk averse and not spending their resources on the many scaled risky bets needed to explore the possibilities. There is more there.
True. I'm a big fan of Cactus's work on the needle family of Simple Attention Network: https://github.com/cactus-compute/needle
That looks like a really nice and well put together library.
But it feels like it sort of backs up my point about there being good models at every size class. Fine tuning Needle looks automated. Yes, you need to know basics like what validation loss means and how to use Python, but otherwise it's all about creating the dataset.
1 reply →