Comment by diggan
11 hours ago
Lets say Meta goes under tomorrow (won't happen, but bear with me) and stops making new Llama releases.
Would the community be able to take over the project and train new models, assuming they have access to the same hardware? Obviously, the community doesn't have access to similar hardware, but even if it did, would the community be able to continue releasing Llama models?
And if the answer to that is no, why is that and how could Llama be considered open source if no one could pick up the torch afterwards (even theoretically), even if they had access to hardware for training?
Its unlikely all the training data for Llama is publicly available, let alone under an open source license. If Llama actually had an open source license (IIRC it doesn't), that would still make it a Toxic Candy model under the Debian Deep Learning Team's Machine Learning policy. That means no-one could replicate it exactly, even if they had the boatloads of cash it would take to buy enough hardware and electricity to do the training. Eventually the community could maybe find or create enough data, but that would be a new different model.
https://salsa.debian.org/deeplearning-team/ml-policy
There are many things to be said about open-source projects and, more broadly, the capabilities of the open-source community.
The most capable parts are for-profit organizations that release open-source software for their business imperative, public benefit companies that write open-source software for ideological reasons but still operate as businesses, and a tiny number of public benefit organizations with unstable cash flow. Most other efforts are unorganized and plagued by bickering.
Llama itself is challenging to take over. The weights are public, but the training data and process is not. It could be evolved, but not fully iterated by anyone else. For a full iteration, the training process and inputs would need to be replicated, with improvements there.
But could another open-source model, as capable as Llama, be produced? Yes. Just like Meta, other companies, such as Google and Microsoft, have the incentive to create a moat around their AI business by offering a free model to the public, one that's just barely under their commercial model's capabilities. That way, no competitor can organically emerge. After all, who would pay for their product if it's inferior to the open-source one? It's a classic barrier to entry in the market - a thing highly sought after by monopolistic companies.
Public benefit companies leading in privacy could develop a model to run offline for privacy purposes, to avoid mass consumer data harvesting. A new open-source ideological project without a stable business could also, in theory, pop up in the same pattern as the Linux project. But these are like unicorns - "one in a million years (maybe)."
So, to answer your question, yes, Llama weights could be evolved; no, an entirely new version cannot be made outside of Meta. Yes, someone else could create such a wholly new open-source model from scratch, and different open-source groups have different incentives. The most likely incentive is monopolistic, to my mind.
I think you've kind of answered a different question. Yes, more LLM models could be created. But specifically Llama? Since it's an open source model, the assumption is that we could (given access to the same compute of course) train one from scratch ourselves, just like we can build our own binaries of open source software.
But this obviously isn't true for Llama, hence the uncertainty if Llama even is open source in the first place. If we cannot create something ourselves (again, given access to compute), how could it possibly be considered open source by anyone?
I understand I was supposed to say “no” and question the open-source label. We’ve heard many arguments that if something can’t be reproduced from scratch, it’s not true open-source.
To me, they sound a bit like “no true Scotsman”. Llama is open source, compared to commercial models with closed weights. Even if it could be more open source.
That’s why I looked at it in a broader sense — what could happen in an open-source world to improve or replace Llama. Much could happen, thanks to Llama’s open nature, actually.
4 replies →
I think the fact that all (good) LLM datasets are full with licensed/pirated material means we'll never really see a decent open source model under the strict definition. Open weight + open source code is really the best we're going to get, so I'm fine with it coopting the term open source even if it doesn't fully apply.
2 replies →
AI models aren't really iterative in the way that other software is. Llama 4 is a completely different product from Llama 3, with different parameter counts and even different modalities. The only reason it gets to be called Llama 4 is that the company that made it is the same and it's convenient to not have to come up with new names all the time, not because there's any sort of continuity with Llama 2.
Fine tunes are the correct analogy to iterative software development—they take the existing code (weights) and improve upon it and modify it—and fine tunes can be produced with what Meta has released.
The bigger problem with Meta's claim that it's open source is that they've attached a bunch of strings to the license that prevent you from using it in a bunch of different ways. It's not open source because it's not open, not because weights aren't source.
No. You need a research lab, compute time and talent to train LLMs.
> No. You need a research lab, compute time and talent to train LLMs.
Right, but even if you had those, could you actually train a Llama model from scratch? You'd still have a lot of work in front of you, compared to a "regular" open source project where you have everything available already, download the source and hit "compile" and you have it done.
And truckloads of data.