Comment by NitpickLawyer
7 hours ago
(from Cursor's blog)
> Training included trillions of tokens of Cursor data which capture a wide-range of user interactions with codebases and software tools. This dataset lets the model learn both from existing software as well as developer-agent interactions, capturing how developers work and how agents interact with their environments.
This is what the big money was for. Cursor is the first big player that had real-world data from real-world projects, before cc / codex were a thing.
> We used reinforcement learning on difficult problems in realistic environments spanning both software engineering and broader knowledge work. These environments teach the model to investigate problems, use tools, recover from mistakes, and verify results.
> Many of these problems had to be designed to be difficult enough that even frontier models fail at them. As models improve, existing tasks stop teaching them anything new, and problems that once required extensive reasoning become routine.
> We developed a distributed agent system to construct these environments at scale. Engineers specify a problem and how a solution is verified, and large groups of agents construct, test, and refine each environment.
This is where scale comes in. You use the previous gen model to prepare datasets for the next model iteration. The better the models, the better the data, the better the next models. (they also have a comparison with their composer2.5 training run, for people still thinking chinese models are "close to SotA"...)
Reports of xAIs demise (after giving a lot of compute to Anthropic) were slightly exaggerated, it seems.
> Grok 4.5 was trained across tens of thousands of NVIDIA GB300 GPUs
Well Microsoft has GitHub and Visual Studio and has no good coding model
Cursor has had a good AI product tons of people used for real work for 2yrs (up until recently when the Claude gap widened significantly) while Microsoft/Github has just been pretending they do with Copilot and awful Github AI integrations nobody likes. Meanwhile Github's code has already been vacuumed up by all the models by now.
you can also use a previous gen model to literally generate data for the next gen model. people used to believe that this is a bad idea but it turns out if you create a scaffold which sinks a lot of compute into generating and grading the data the quality turns out great.
well the big money was also in spacex stock, fresh post IPO, so overall a very smart move it seems
> You use the previous gen model to prepare datasets for the next model iteration
I've read multiple times that this approach is harmful in training.
You're essentially describing what many call distillation, but it's only useful in post training to guide behavior, it teaches how to behave, not how to think.
I might be wrong though and would be glad if someone more knowledgeable provided more insights.
There have been papers about model collapse, but the underlying assumption is that you constantly train on only the outputs of the previous model. Later research has shown that as long as you retain some "real" data, training on largely synthetic data is ok.
And in the case the previous poster describes, the other model doesn't generate datasets, it generates environments which the next generation interact with to learn from.