Comment by conradkay
6 days ago
Wow, seems worse even on price/performance than GLM 5.2, which is only 744b parameters.
From the system card: "On CyberGym vulnerability discovery, Claude Sonnet 5 is less capable than Sonnet 4.6, and far less capable than Opus 4.8 and Mythos 5
As with the other evaluations in this section, these results were achieved with all safeguards turned off. When run with our default mitigations, Sonnet 5 scored a 0 on CyberGym"
I have tried to rewrite an article with GLM-5.2 and with Sonnet 4.6. Completely different results as LLM is non-deterministic. But GLM-5.2 made a lot of subtle mistakes that needed to be corrected by hand. On the opposite, Sonnet found and corrected all mistakes in the second round.
Similar situation was with planning and coding. GLM-5.2 seems to be good “on paper” but the real usage results was different.
And I am not an attorney for Claude or GLM-5.2… :)
But as I’ve been using LLM models daily since Nov 2022 I have realized that all common tests have to be confirmed in your project - there is no “one model rules them all” - you need to dig out a specific model from that LLM haystack with thousands of models.
Benchmarks help but they start to be similar to fuel consumption specs in car ads - real consumption is different for everybody :)
> Completely different results as LLM is non-deterministic.
You'd need to produce this like 20 times by each model and then do 2x20x20 cross comparisons by both models and ultimately distill the 2x20x20 comparison results into two reports of how they differ.
In this non deterministic computing future, everything else is voodoo, feelings and "vibes".
I would expect a model's result each time to be of a similar quality to the other times. There's something wrong if it does a way better or worse job, at the same problem, sometimes. It's possible, but I haven't heard anyone saying that they do.
2 replies →
I’m just dipping my feet in the water of local models and I really feel this. I had a simple alignment task (align known quality transcript without timestamps with timestamped but lower quality whisper transcriptions) and I went through 12 rounds of testing across 4 generations of 3 models. The results were all over the map even across versions of the same model. Spin out the task to something as big as coding and wow.
If you have any advice/blogs on doing project specific benchmarks I’d love to hear it. I’m trying, but it’s haphazard at the moment
We have found similar when plugging GLM 5.2 into actual benchmarks in our product. The open-source models are really dialled into the public benchmarks, until you try them in context you won't have a solid idea of how they perform (Sonnet is a higher quality model than 5.2, both in prose, reasoning, and alignment).
Probably because benchmarks are leaked. Included in their training or model can cheat by finding answer online.
Most benchmarks notoriously undercount multi-turn instruction following too. And it's where open source models (and Gemini) lose big time.
Finally, a viable business strategy - sell security-oblivious code monkeys for cheap, then charge premium rates for agents capable of cleaning up the mess.
I think instead they should sell super hackers and get their product banned instantly and go bankrupt
Judging by the events of recent weeks, I'm guessing the low cyber results are why they were allowed to release it
Not to single you out, parent commenter, but I really hope the quality of discourse on HN will move past these basic comparisons eventually. It seems like every thread on every model release has the exact same comments.
"Wow, X models is Y% better or worse than Claude Z model on T benchmark"
"That's irrelevant, they're just benchmaxing."
"Not useable for daily coding or agentic workloads, the vibes are totally wrong."
"It's almost as good, and costs a lot less, so I will absolutely use it."
"I cannot imagine justifying using these, as the step change means open models lower costs do not make up for the productivity loss"
I'm an unhappy Anthropic customer and really rooting for open models and non-gatekept intelligence, but how do we move on from this now meme-like model release discourse rigamarole. I do not know what that would be. I don't design LLMs nor benchmarks, and I genuinely appreciate that people do their best to provide information, even if non-perfect here. I'm sure most of you who actively read these comment pages on announcements must feel similarly, though, right?
I'm not sure what else can be said? I've found benchmarks to be a very weak signal for how good/bad the model is, but it's the #1 thing the companies highlight.
20 minutes after the announcement there's no real useful statement that can be made about it.
Yeah you definitely have to be skeptical regarding sentiment for open/local model capabilities, since there's bias from what people want to be true.
I generally agree with this in spirit https://www.seangoedecke.com/are-new-models-good/ , but I think you can read Anthropic's results showing Sonnet 5 as almost strictly worse than Opus 4.8 as very credible/meaningful, and then draw comparisons from that
I feel the same way sometimes.
I read a comment earlier that said "I think it's likely that they've scraped all the code regardless of license and trained on it, given how much they scrape the web."
That's what every other comment said like 3 years ago. Where has this guy been?
The trends in discussion about LLMs gets very, very tired--there's little added but personal opinions.
"It's totally obvious they quantitized Claude Z"
At least we quit with the "i asked it this question and here's what it said" comments. They were truly awful for the first 6 months or so.
Or the "I have my own personal benchmark..."
"Claude and its political bias thinks the supreme court should..."
2 replies →