Comment by grunder_advice
10 hours ago
Whenever these kind of articles pop up, I always think how sad it is that PyTorch, Llama and many widely used opens source projects are tied to Meta.
10 hours ago
Whenever these kind of articles pop up, I always think how sad it is that PyTorch, Llama and many widely used opens source projects are tied to Meta.
They are open-source. Shouldn’t we be happy that at least something good comes of that sentient pile of cash?
So get a group of other sympathetic people and fork them.
This is virtually the only place where you have a chance to take power from them by your actions.
"The best way to complain is to create things," and yes that's a poster I got for free back when I worked at Facebook.
> fork them
This requires all of the "source" to be available. For PyTorch and a bunch of other projects, this is trivial as all the source is straight up on GitHub. But for proprietary things like Llama, it's really hard to fork something when you don't even have access to what they used to build it (software-wise, not even thinking about the hardware yet).
How could you fork something like Llama when Meta don't even speak clearly about what data they used, literally none of the training code is available, and you have to agree to terms and conditions before you're "allowed" to do anything with it?
> you have to agree to terms and conditions before you're "allowed" to do anything with it
I don’t have experience with this so I’m taking it at face value; if this is true, it’s so strange that I have an idea of this being an “open” model. As in, not that they PR’ed to make people believe it but that people who were required to accept those terms seem to believe it (as users seem to repeat it). Seems a little bit of critical thinking should dispel that notion. Are there any, more reasonably open models? Is LLaMa just called open because it’s the most accessible?
1 reply →
Be thankful they are open source at all. See OpenAI for the alternative.
Lets say Meta goes under tomorrow (won't happen, but bear with me) and stops making new Llama releases.
Would the community be able to take over the project and train new models, assuming they have access to the same hardware? Obviously, the community doesn't have access to similar hardware, but even if it did, would the community be able to continue releasing Llama models?
And if the answer to that is no, why is that and how could Llama be considered open source if no one could pick up the torch afterwards (even theoretically), even if they had access to hardware for training?
Its unlikely all the training data for Llama is publicly available, let alone under an open source license. If Llama actually had an open source license (IIRC it doesn't), that would still make it a Toxic Candy model under the Debian Deep Learning Team's Machine Learning policy. That means no-one could replicate it exactly, even if they had the boatloads of cash it would take to buy enough hardware and electricity to do the training. Eventually the community could maybe find or create enough data, but that would be a new different model.
https://salsa.debian.org/deeplearning-team/ml-policy
There are many things to be said about open-source projects and, more broadly, the capabilities of the open-source community.
The most capable parts are for-profit organizations that release open-source software for their business imperative, public benefit companies that write open-source software for ideological reasons but still operate as businesses, and a tiny number of public benefit organizations with unstable cash flow. Most other efforts are unorganized and plagued by bickering.
Llama itself is challenging to take over. The weights are public, but the training data and process is not. It could be evolved, but not fully iterated by anyone else. For a full iteration, the training process and inputs would need to be replicated, with improvements there.
But could another open-source model, as capable as Llama, be produced? Yes. Just like Meta, other companies, such as Google and Microsoft, have the incentive to create a moat around their AI business by offering a free model to the public, one that's just barely under their commercial model's capabilities. That way, no competitor can organically emerge. After all, who would pay for their product if it's inferior to the open-source one? It's a classic barrier to entry in the market - a thing highly sought after by monopolistic companies.
Public benefit companies leading in privacy could develop a model to run offline for privacy purposes, to avoid mass consumer data harvesting. A new open-source ideological project without a stable business could also, in theory, pop up in the same pattern as the Linux project. But these are like unicorns - "one in a million years (maybe)."
So, to answer your question, yes, Llama weights could be evolved; no, an entirely new version cannot be made outside of Meta. Yes, someone else could create such a wholly new open-source model from scratch, and different open-source groups have different incentives. The most likely incentive is monopolistic, to my mind.
I think you've kind of answered a different question. Yes, more LLM models could be created. But specifically Llama? Since it's an open source model, the assumption is that we could (given access to the same compute of course) train one from scratch ourselves, just like we can build our own binaries of open source software.
But this obviously isn't true for Llama, hence the uncertainty if Llama even is open source in the first place. If we cannot create something ourselves (again, given access to compute), how could it possibly be considered open source by anyone?
8 replies →
AI models aren't really iterative in the way that other software is. Llama 4 is a completely different product from Llama 3, with different parameter counts and even different modalities. The only reason it gets to be called Llama 4 is that the company that made it is the same and it's convenient to not have to come up with new names all the time, not because there's any sort of continuity with Llama 2.
Fine tunes are the correct analogy to iterative software development—they take the existing code (weights) and improve upon it and modify it—and fine tunes can be produced with what Meta has released.
The bigger problem with Meta's claim that it's open source is that they've attached a bunch of strings to the license that prevent you from using it in a bunch of different ways. It's not open source because it's not open, not because weights aren't source.
No. You need a research lab, compute time and talent to train LLMs.
> No. You need a research lab, compute time and talent to train LLMs.
Right, but even if you had those, could you actually train a Llama model from scratch? You'd still have a lot of work in front of you, compared to a "regular" open source project where you have everything available already, download the source and hit "compile" and you have it done.
And truckloads of data.