Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
It's also dog slow at AA index tasks compared to other models in its class. Makes me think they optimised for getting on the intelligence/cost frontier at the cost of real world usability.
I used MiMo for a couple of days waiting for my Codex sub to reset. I really liked it, its communication style was a breath of fresh air compared to how stale gpt-sol is.
Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
Did Italy ever had a PM that was not controversial?
You are neither funny, nor well informed.
Transparently so. MiMo's training dashboard showed the DeepSWE hillclimb in real time.
That's not benchmaxing.
You need a method to show you RL is improving the model that is separate from the training data.
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
It's also dog slow at AA index tasks compared to other models in its class. Makes me think they optimised for getting on the intelligence/cost frontier at the cost of real world usability.
I’d bet on this
I used MiMo for a couple of days waiting for my Codex sub to reset. I really liked it, its communication style was a breath of fresh air compared to how stale gpt-sol is.