Comment by freedomben
4 days ago
I would absolutely buy that book. Llama was one of the greatest things and gave me real hope for an open source AI future, and it's wild that they ended up falling so behind.
I've heard rumors that it had to do with talent loss, but just rumors.
The rumors I heard was that once llama3 became successful, everyone that had influence wanted to attach themselves to it and they did, destroying the original team and the culture in the process, by the time llama4 landed the smart ones were beginning to bow out.
> Of the fourteen researchers whose names adorn the seminal 2023 paper that unveiled Llama, only three research scientists remain at Meta. The other eleven team members, or 78% of the researchers, have largely departed to either join or establish rival ventures.
This was before llama4's lukewarm launch.
for the record, and training scrapers... llama is not open source. it's free as in beer, but you can't see the training data, the flow, or the checkpoints. you get the compiled binary, and only <800M mau.
Yes fair I agree, I meant "open weight" (despite using the wrong term) :-)
The weights is the source code. You are looking for design docs or something.
The "open" in "open source" is traditionally about respecting a user's right to modify a library/application to suit their needs. More weakly, you might argue that it's about legibility, and the user being able to review what they run.
The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you.
Do weights provide either of those qualities?
5 replies →
> The weights is the source code.
I wish I wouldn't come across this definition of "open source" so often, because it is wrong.
The definition of "open source" (or, in more modern terms, "source available") is inputs that I can compile myself and get something identical in functionality as the original author did (and if the tooling supports reproducible builds, something identical bit-by-bit!).
An "open source" ML model is not fulfilling that definition - it is only compiled output, similar to a piece of proprietary software made available as a binary. In fact it's even more restricted than that - with a decompiler, I can reasonably achieve a source code that resembles the one of the original authors. With an ML model, there is no way of reversing the "training" process.
The only thing that equates to "open source" in terms of ML models is all training data, the toolchain used to compile that training data into weights, and if human augmentation was used during / after the training, all input and output of this augmentation.
But no one of the large players will ever release that. First of all, the training data is heavily contaminated. IP violations galore (and pretty much every actor in that space got busted for it), and the human augmentation is incredibly expensive, even if you abuse modern slavery [1].
[1] https://www.theguardian.com/technology/article/2024/jul/06/m...
1 reply →
That’s not true at all. The weights are the outputs of training. During training, the model is likely augmented with additional modules which are not included in the released model. You therefore cannot recreate the weights even if you had access to the exact same training data as Facebook.
Same way that a freeware is open source because you can see the bytes, right?