Comment by HarHarVeryFunny

I like that Alibaba are emphasizing more challenging real-world "coding" tasks as well as just benchmarks.

The "Reproduce a research paper — then improve it" seems particularly impressive as well as actually useful - perhaps as close as we're going to get to "recursive self-improvement" given that these are data-driven not code-driven models.

I'm not sure how much real world use of AI is 100% hands off agentic coding, but I guess for objective evaluation purposes it needs to be something without a human in the loop.

It'd be interesting to see a comparison of each company's best models on a suite of real-world challenging tasks like this, but I guess difficult since each group of tasks would need to be one-time-use only to prevent subsequent benchmaxxing which makes comparisons useless.