← Back to context

Comment by Aurornis

4 hours ago

I doubt that would change the perception. Every model release is followed by accusations of nerfing.

There are several projects that repeat benchmarks on published models. None has ever found significant fluctations

Here's one example https://marginlab.ai/trackers/claude-code/

Fluctuations of a few percentage points are to be expected and should not surprise anyone who knows how LLMs work.

This Twitter analysis of Fable 5 is not that at all. They analyzed their coding sessions and blamed all of the fluctuations on Fable changing. They then compared to ARC-AGI-2 questions as the benchmark for thinking tokens and tried to stir up anger that coding turns don't produce as many thinking tokens as the ARC-AGI-2 problems.

8 comments

Aurornis

Reply
ricardobeat  4 hours ago

This page has been in 'New model — collecting baseline data. Degradation detection paused.' state for months now. It seems to never say 'degraded'.

If you look at the graphs, the latest benchmarks are showing a pretty significant dip, and they match pretty well with some horrible experiences I've had in recent weeks. You can see token usage steadily going down, matching exactly what the author measured on his own.

  • Aurornis  4 hours ago

    > This page has been in 'New model — collecting baseline data. Degradation detection paused.' state for months now. It seems to never say 'degraded'.

    Click the part at the end that says "View historical performance". They wait to collect more data about a new model before adding it to the overall charts.

    The overall solution rate continues to climb when new models are considered.

    > If you look at the graphs, the latest benchmarks are showing a pretty significant dip, and they match pretty well with some horrible experiences I've had in recent weeks. You can see token usage steadily going down, matching exactly what the author measured on his own.

    The y-axis is amplified to make differences look larger than they are.

    Hover over the dots to see the confidence interval. A 1-2% change means nothing.

    • ricardobeat  4 hours ago

      The page is currently tracking Opus 5, which was released July 24 and has not seen any updates since.

      30-day average 83% [71-91], last result is 79% [66-88], and it dipped to 75% [61-85] a week ago.

      3 replies →

jesse_dot_id  1 hour ago

It would change my perception but only if there were a competent and stringent administration in place. I didn't used to have to wonder if the ground beef I was buying was actually 1lb because there were inspections and repercussions, but stuff is kind of chronically underweight these days.

A properly run OWM enables you to stop wondering if you're being ripped off and that's what AI needs because I think it's incredibly easy to just assume we're being ripped off because these companies are all built on a foundation of wonton theft. (Not that I really care about that — I think all information should be free, but still.)