Comment by esskay
2 days ago
I'd be interested to know what was going on with it during the public test as there were numerous reports of it improving considerably at tasks it was asked to do early on in the test compared to later in it.
2 days ago
I'd be interested to know what was going on with it during the public test as there were numerous reports of it improving considerably at tasks it was asked to do early on in the test compared to later in it.
It's logical to serve the best version (quant) of the model at the beginning so that users keep testing it. It is also reasonable to think that the developer of the model tried to test various quant levels by gradually degrading the model's capabilities.
I mean that’s imaginative but not sure there’s any evidence at all for it, and it’s the opposite of what the comment you replied to observed.
Two potentials from my pov:
1. Just variance in pass@K. If you prompt any model multiple times you'll see a large variance. N=1, but I find chinese open source models have a higher variance than higher-RL'd models like fable/opus.
2. They legitimately shipped a new RL checkpoint over the 7 days, which I find hard to believe.
I am leaning towards 1.
Or 3, they find some bug/regression in their pipeline; maybe they didn't quant parts of a model properly, maybe their inference engine had a bug, maybe some pinned MoE expert wasn't pinned, etc...
That's very plausible to have, identify, and fix in a day; especially when you get community feedback in the wild.
3. Deployment problems unrelated to the weights causing degraded performance
For sure the version accessible from OpenCode had a massive timeout problem the first day or so, which seemed to heavily degrade its task completion rate
2. There was a new checkpoint. Official.
do you have reference to where it was said?
lol don't shout out the obvious