Comment by a11r
3 days ago
Artificial analysis shows this: underperforming Qwen3.8 Flash Next and GLM 5.3 Flash, both of which you can run on a single B300 . I think Meta has a bit of catching up to do in the open-weight space.
As an aside, it's really unfortunate that artificial analysis only provides comparison at the highest reasoning level, which is mostly unusable for high-volume production use cases. I would really like them to start providing comparisons at low and medium reasoning levels across these models because those tend to be the ones that actually get used in production.
Been using GLM 5.3 Flash via Ollama Cloud basic subscription:
if this is the only model I had to use for the rest of my life, I would be cool with that. It's good at a wide variety of tasks including reverse engineering, design, working with ESP 32 etc.
I think Muse Spark 1.3 is better but GLM 5.3 Flash is an excellent daily driver.
It is unreal how good GLM 5.3 Flash is. There's an ~4 bpw EXL3 quant that fits on 2x DGX Sparks. I'm not sure it's worth it now ($7k/Spark) or even when I bought them ($4k/Spark) but there's something about knowing that the whole internet could go down and I could still have this model running at 50-60 tokens/s with 1M context, indefinitely. It's just so good. TensorFold hits 100+ tokens/s in aggregate with 2-3 M total context pool. It's like having Claude at home.
I'm using Flash and the full 5.3 model side-by-side, and for coding I'm hard-pressed to find a difference. Flash is slightly worse at steering in longer contexts, but writes code that's just as good as anything I get out of full 5.3. You could totally build a SaaS or video game with nothing but 5.3 Flash.
The full 5.3 is only really needed for super deep bugfixes/profiling passes, or ridiculous reverse-engineering in my experience.
3 replies →
Am I the only one experiencing accidental chinese tokens, wrong grammar, and other sloppy mistakes in almost every output when using GLM 5.3? I feel like using it through any kind of subscription service (z.ai, Opencode Go) just serves me aggressive quants.
On some days it's usable, but on most days it feels more like having to trust a pretty smart intern that may or may not get a small stroke every couple of minutes.
7 replies →
How do you feel it compares to DeepSeek 0731 if you have tried that? (Also runs well on 2x spark)
2 replies →
How did you setup the 2x dgx spark?
1 reply →
I've been using GLM 5.3 Flash as the planner and DS 4.1 as the task worker over the last month and the results have been very good, predictable, ridiculously cheap.
Why this model split? And are your work volumes so high that you prefer these cheaper models over a subscription or are you simply trying to avoid the 200/m? (which will only go up).
What reasoning level do you use for each?
Genuinely curious, why not use GPT Sol 6.1 high which has a better cost per task and intelligence ($0.32, 50) or Mimo 2.6 pro? ($0.13, 46) over GLM Max's $2.01, 45?
Sol won't do some things. Like, GPT freaked out yesterday because I needed to vector trace a logo and it said it would be a rights violation.
Then I have to work down the list: Grok, then Muse, then if those fail I make the Chinese one do it.
https://imgur.com/a/8LVmpLV
21 replies →
Not OP, but I've been trying out Sol 6.1 medium with hermes agent, and I have not been able to match usage costs to what I've been getting with glm 5.3 flash high.
Really don't know why, since I've offloaded thinking to the LLMs in this area of my life. Though even at medium, 6.1-Sol is noticeably better at most tasks I throw at it.
My experience has been that the "cost per task" estimate wildly underestimates the cost of top models on most real work, because the tasks (as I understand it) tend to be isolated bits of problem solving or code generation, which rewards writing code that works the first time and not generating too many reasoning tokens. Sol is great for both. However, a lot of real work involves tracking down bugs, working through tests, examining many parts of a codebase, refactoring... even with subagents and good tool use this tends to bloat context before the problem gets solved. Deepseek's absurdly low cache price really shines in these instances, but even GLM 5.3 Flash easily beats the top models on price in real world usage. Maybe that's different if you factor in subscriptions.
I'm OP and I have the Ollama basic $20/mo plan.
Been very happy with it although I will say Muse Spark 1.3 being free on OpenCode is my main model these days. If I run out of free credits on OC (which almost never happens) then I drop into GLM Flash 5.3.
Because it's not as good IRL. I find that DS Flash does just as good of a job on most things.
If you liked GLM 5.3, you would appreciate Deepseek V4 Flash a lot. In my personal opinion and use-cases, it outperforms GLM 5.3.
V4 isn't the newest DeepSeek, and GLM 5.3 non-Flash isn't the same ballpark
- DeepSeek V4.1 Flash has marginally better coding, and smaller experts (8B-In/16B-Out) with more compressed attention and KV cache, so it's faster but will have less expertise per task. It's a 552B model, so while it theoretically may contain more trained data it also requires more RAM (the experts are smaller to compensate for speed loss) making it harder to serve without the distributed KV cache DeepSeek hosts.
- GLM 5.3 Flash has marginally better agentic work, and has video input (DS doesn't), and its experts are 18B so there's more intelligence per task. It's a 320B model so it requires less RAM which will make it cheaper to serve. Considering it's a much smaller model, the higher intelligence indicates their architecture/training is better.
- GLM 5.3 (non-Flash) is far more intelligent than both; it's a 744B model with 40B experts, but it's text-only.
3 replies →
I'm the weirdo who is clinging onto GLM 5.2. 5.3 seems to have picked up the bad writing from Opus 5.
GLM 5.2 was the first open model that I was happy to use as my main. It's a fantastic model. 5.3 cost a lot more per task, was slower and didn't get that much better results for me, so I tried 5.3 Flash for a bit, it was ok, but didn't feel as magical as 5.2 and I have now landed on DeepSeek 4.1 Flash as my main.
It's not just you-- the claudisms in 5.3 are pathetic and drive me crazy, I was very sad to see it go in that direction and consider it a big step down in that respect.
BUT OTOH the performance of 5.3-flash are impossible to argue with-- it has the same pathetic claude behavior full of "Honest" and "lever" and "load bearing" tripe... but it's scortchingly fast on hardware that is pretty slow running 5.2/5.3.
As a result I use 5.3-flash a lot, in spite of feeling kind of gross about its language. I don't enjoy using it the way I enjoyed GLM-5.2, but its pretty effective.
I don't think that's likely, given the close release dates of both? I wonder if 5.3 just choked on too much RL slop, independently of Opus 5.
Stupid question probably because I’ve gotten out of sync with the new releases but is Flash better or worse than vanilla 5.3? I’ve been using 5.3 extensively (and maybe ironically have used flash only for viewing images which I have as a skill in Pi because 5.3 vanilla seemingly doesn’t support vision). I find it generally competent but still very far behind opus 5.5. I’ve been trying to avoid Anthropic but the lift of opus vs GLM is enough that I end up coming back to it. I feel like currently there’s no comparison.
anyway, tldr I would not be happy if stuck with vanilla 5.3 so is flash better and should I move to that?
GLM 5.3 Flash is better if you need native multi-modal capabilities, higher speed, and lower costs, but worse than vanilla 5.3 in raw reasoning and for complex programming.
Have you compared to deepseek v4.1 flash? It’s so good, at a ridiculous price
I agree, however It's too easy to game arbitrary limits like "medium". There is no olympics for "just rolled out of bed and slightly hungover"
We don't even need a standard for reasoning levels.
Just accept whatever the provider names as "medium" and compare the accuracy between models vs the time it takes to complete a task.
The fashion industry has been doing this for decades. Medium means different things for different brands.
You don't need to directly compare the same reasoning levels. Today you can compare Sol on Low vs Haiku on Max, for example, but you can't look at GLM 5.3 High, only Max.
For some tasks I genuinely feel Opus 5.5 does better on low than medium or high. The trouble, as always, is figuring out which.
Edit: I realize after commenting that you're not suggesting medium is useless, but just arbitrary and differs between models. Personally I don't want to directly compare mediums between models, I just want to be able to compare across effort levels.
i run opus on medium, sonet on high, and haiku on high for implementation based how complicated is for planing and specing and review opus is on high usually, except if if the task is contained.
did not try to run it on low, will have to try it.
2 replies →
That's the beauty of the line charts; the labels are arbitrary points on the line. You can easily compare the series visually.
A graph of time (or cost) per task against success rate does it. Just include all the different levels.
This exactly. The three interesting parameters are total time to answer, total cost, and output quality (for any given purpose).
[flagged]
> As an aside, it's really unfortunate that artificial analysis only provides comparison at the highest reasoning level
They do other reasoning levels for some models.
Here is Sonnet 5.5 across the full range of efforts from low to max: https://artificialanalysis.ai/?models=claude-sonnet-5-5-low%...
Opus 5.5 low to max: https://artificialanalysis.ai/?models=claude-opus-5-5%2Cclau...
Here is Muse Spark 1.3 Max and Xhigh: https://artificialanalysis.ai/?models=muse-spark-1-3%2Cmuse-...
I'm guessing it depends how much the vendor wants to contribute. Running these benchmarks isn't cheap!
The most important graph for me on Artificial Analysis is this: https://artificialanalysis.ai/?agentic-speed=intelligence-vs...
It shows the time to complete a task versus intelligence index, which is much more important for my use-case as I do quick back and forths usually. Someone on here suggested it to me a while ago. It also captures speed and token efficiency nicely, since both of these decrease time per task.
Yep, just benchmarked it against GLM5.3 Flash on my own local bench and it's slower in TTFT, tok/s and concurrency. It also scored lower.
Tried different reasoning levels and didn't really see any difference but GLM 5.3 flash only really works high, max or none so I couldn't really compare the lower efforts which appear to work on Spark.
this is properly bad as you can run glm on 6000pro, at (in my eyes) pretty good quant, and a q2 on a 5090. im suprised at how good 5.3 flash is at some parts. i still think ds4.1 flash is better but it is more demanding. i hope meta pushes something in a range of 300-400b with a 15-20b active parameters. i know its a crowded space. but it would be fast and probably mostly good enough.
Have you tried two 6000 Pro Blackwells with a model spread between both ?
> As an aside, it's really unfortunate that artificial analysis only provides comparison at the highest reasoning level, which is mostly unusable for high-volume production use cases. I would really like them to start providing comparisons at low and medium reasoning levels across these models because those tend to be the ones that actually get used in production.
The frustrating thing is they started doing this for the big labs—you can see their benchmarks for all reasoning levels of Sol 6.1 or Haiku 5.5. They just don't seem to do it for any of the open weight models.
I feel like reasoning is a useless metric, and we should instead focus on results, and time to achieved said results. That is a more useful metric overall.
Reasoning level influences both of those things dramatically. So we need to have it available as a parameter for comparisons.
not a metric, it's a parameter used at inference time
Yes, but what metric defines what constitutes a given reasoning level? That's what's arbitrary here, not the label (which is also arbitrary).
There’s a reason they put the “US” in “the most capable US open weight model”.
Model X vs Y is the new Vim vs Emacs.
> Artificial analysis shows this: underperforming Qwen3.8 Flash Next and GLM 5.3 Flash, both of which you can run on a single B300
You don't need anywhere near a B300 to run Qwen 3.8-Flash-Next in its Q8 largest variant published as a GGUF by unsloth. It fits very comfortably in a 256GB system.
> only provides comparison at the highest reasoning level, which is mostly unusable for high-volume production use cases
Similarly, AFAIK evals tend to be performed at native precision, while anyone who cares about speed is probably running quantized.
My immediate reaction is that it’s weaker on coding than other open-weight models and that coding is a primary use case for running locally (not the only one, but surely a major one). That said, it takes a lot to get full Kimi and GLM running locally.
Qwen 3.8 flash next runs pretty damn fast on my 5090, no need for a b300.
Yes, running Qwen 3.8 flash next on a 5090 with 96GB of RAM, using Strata. Fast and good.
Artificial analysis puts it next to Mistral's new model.
Which is crazy bc they promised to open source muse 1.2 which is good. And they give us this bait and switch
yeah for botsbench.com side where we do cybersecurity investigation evals of models + harnesses, we've been having to play with:
1) time budget model gets (5min / task) -- this became a real problem with opus/astra/fable at high+
2) tuning which effort level we report on (when not all)
Why didnt thry include OpenAI or Anthropic models? I get they were comparing open weights.
very true. when configuring custom agents for enterprise, we almost always default to none or low reasoning knobs. The added latency from medium or higher reasoning levels compared to the improvement in response quality is a diminished return.
it’s quite suspect to claim that a model is ‘underperforming’ so resoundingly, especially after so little time since release.
there’s this objectivist approach of reducing llm quality to a single dimension, furthermore one with measurability bias.
my thesis is that the chinese models beat the american models on benchmarks, on quantitative measures, but probably do worse in subjective measures. to put an example, compare a chinese phone with an iphone, the specs are great, but the OS? the materials? the privacy posture?
so will chinese llms have great specs and benchmarks, but will miss the mark when it comes to the actual quality in practice when your input stops looking like the benchmarks it was trained on.
America number one, it’s the chinese that play catch up with clones, not the other way around
Ahat's the cost of a b300 versus two mac minis?
How much longer does it take to ask this and wait for response compared to checking yourself?
I'm making a point.
1 reply →
Um a 512GB m5 max Mac Studio is what 25k? A B300 GPU and the computer to use it is what? Double? Triple that? Apples v oranges my friend. Apples v oranges.
[flagged]
[dead]
[dead]
AA is a joke.