Comment by breckenedge

6 hours ago

Do these evaluations get re run a few weeks after launch? I started doing that yesterday for our internal dataset and found Sol’s performance had regressed to be equal to Luna’s. Granted this was one run, but something I’m becoming more concerned about, the model providers want to quickly prove they’re the best, people switch to them, then they pull the rug.

6 comments

breckenedge

Reply
tedsanders  1 hour ago

Can you share more information on your methodology?

GPT-5.6 Sol's performance in the API should not change over time. If it has, that's a severe bug and we'll look into it.

We do sometimes tweak ChatGPT settings (e.g., tools, system prompts, efforts) over time, but we never play games to juice evals at launch times. You should always get what's advertised.

(I work at OpenAI.)

  • breckenedge  28 minutes ago

    This is all code review runs via OpenRouter with a Pi harness, and it’s totally possible there are shenanigans going on elsewhere.

    Yesterday, I ran an identical bug identification dataset from two weeks ago, saw a 50% drop from a few weeks ago, putting Sol on the same level as Luna. Sol had been finding 40-50 bugs per set, then dropped to 25, matching Luna’s performance. Not enough to establish a pattern, but enough to raise eyebrows.

    Our review workflow is public if you want to peruse it, dataset isn’t. The process isn’t really stabilized yet either as I have to balance running this against limited budgets.

    https://github.com/BiggerPockets/.github/blob/main/.github/w...

    • tedsanders  22 minutes ago

      How many tasks were in this dataset?

      If it's a single task where it dropped from 50 to 25, it could be random variation (not saying it is, but it could be). If it's the mean over hundreds of tasks, that suggests a problem with either the eval code/harness or our API.

echelon  4 hours ago

These tests need to be sampled continuously.

Moreover, the tests should be randomized somehow to ensure the models don't memorize the answer.