Comment by Almondsetat
12 hours ago
Sorry, where exactly is "data" in "source"?
I can understand not wanting to call it open source if they don't give you the algorithm and software used for training, but wanting the training data too? That's completely different
For LLMs the training data is the source of their weights.
You can‘t reproduce the LLM without the same data
That’s not really true. The models have source code defining their architecture and it is open-source.
People keep trying to shoehorn OSS concepts onto model weights, but the concepts don’t fit because the weights aren’t software. They aren’t compiled code. They are learned parameters to use with a (very big) function that itself is expressed in the code.
So they’re a very valuable asset that complements the code, but they are not the code. You could use randomly initialized weights and the software will work - it will output tokens. They just won’t have useful patterns.
I don’t think OSS definitions have ever required that assets have their source included. For example, artwork is very important to a game, but nobody thinks a game is not open source if it doesn’t come with sketches and a copy of Adobe Illustrator to recreate the artwork from scratch.
Edit: The distinction I would draw between models like Deepseek and models like OLMo is whether they are open science. With a model like OLMo, they have published everything you need to replicate the training experiment. Whereas Deepseek does share a lot of knowledge, but keeps a lot proprietary too.
Your game example is a great one: If a game didn’t come with its 3D models, sound, level files, game data, or assets of any kind, but its engine was open source, I don’t think anyone would say “hey this game is open source”. They’d say the engine is, but that’s it. All of the above things are not code, but they are part of a game and yes, expected to be included in an open source game.
8 replies →
My understanding was that there was a lot of non determinism in the training process because of many factors: how GPUs optimise floating point arithmetic, how the compute is distributed between the thousands of GPUs etc
I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things).
You'd likely get a model with very similar behaviour but the weights would be different.
Please someone correct me if I'm wrong.