Comment by efficient_dairy
20 hours ago
I don't find open-weight models that impressive anymore. MiMo-V2.6 already showed that you can have a not so crazy architecture and enough compute, the bottleneck is then just the data. OAI and Anthropic are largely the frontier models because of the synthetic data they made. They have a large customer base and have the user's data as well as knowing what tasks their customers use the models for and what domain they should get synthetic data for.
Interesting. Have heard about the need to create synthetic data for LLM's, but didn't realize it was such an important factor. Although, how big a factor synthetic data is the next question, but guess there is no way to truly verify how much difference it makes with these closed models.
This is literally what the labs have been doing over the past year. The whole idea of emergence is a lie, there is some interpolation and superhuman long evaluations the models can do, but almost all the gains are from synthetic data. They hire thousands of professionals and pay them as much as 200$/hour to create many tasks that they want the model to perform and use these to teach the model on how to do it with RL. OAI had 30,000 contractors from Mercor for Sol 5.6. When you see Opus suddenly becoming great at blender or some other 3D graphics, that's because they hired professionals and had them do similar tasks that people are looking for. They keep having better professionals at each iteration and so the quality improves. There is no emergence or "General" intelligence. The model doesn't learn to become better at a task because of scaling laws or emergence or whatever they might wanna say, it is literally RL on tasks that they want the model to perform well on.
Also interesting. Do you happen to know if once they've used those thousands of contractors to teach the model something like Blender (or some similar app) using RL, when they training a new model, do they need to use same contractors again to teach that new model the same behaviors?
The reason I ask is even though it can take hundreds or thousands of contractors to teach a model a certain behavior, wonder if they really only need to do it once for each desired behavior? (of course, future models might expand and refine this previous training) Because if that is the case, then wow, then future models can really expand their capabilities really very fast.
...Am wondering if they somehow record a training session so they can play it back whenever they need to train a new model with the same info? Or maybe the new models can just use distillation from the old model to relearn the old behaviors?
1 reply →
What you're talking about is not synthetic data. Synthetic data is data generated by an LLM. If the data is generated by a human, it's not synthetic.
You're talking about real data created and curated by humans to help in training LLMs.
1 reply →