Comment by aliljet
3 days ago
This is absolutely still shy of Sol and Fable, but only just by a hair. Ridiculous results. There's still not a compelling economic reason to drop OpenAI courtesy of the ludicrous reset addiction that's taken place, but it feels like we're on the precipice.
How are you all toying with running this kind of thing in a mega quantized way locally? Two weeks out from released weights, but this is still just GLM 5.2 with post-training magic.
OpenAI and Anthropic are both seeking trillion IPOs, while Chinese labs are pumping out open-weight models that are free for US providers to host and monetize.
These Chinese models cost less of US SOTA models to run, even if they are less capable. Providers can just run them, offer cheap tokens, and pocket the margin.
I just don't see how you justify a trillion valuation for US AI labs when the underlying models are being commoditized this fast.
This is going to be catastrophic.
Whether AI works or is useful or not isn’t even the question anymore. It can fulfil every promise Sam Altman has been making and will still make no financial sense to justify these valuations.
I take it from [1] (transcript of recent DeepSeek CEO discussion with investors) that DeepSeek would disagree on the immediate catastrophic impact to the likes of OpenAI or Anthropic. The reason is even though technology parity mostly exists, only OpenAI, Anthropic et al have the inference capacity to gain market share and generate revenue. Chinese vendors don't have the chips needed to scale up inference and gain market share, and the DeepSeek CEO doesn't think this would happen in optimistic circumstances in the next 3 years, but thinks it might be possible in 5 years.
In summary, regardless of country of origin, availability of inference capacity is the moat protecting the likes of OpenAI and Anthropic, not technology superiority.
[1] https://www.fredgao.com/p/deepseeks-liang-wenfeng-breaks-his
16 replies →
I have already begun winding down my spend on claude and OAI to make room for infra budget. Anecdotal, but I have no doubt a lot of others are doing the same, I very much agree the US players have major issues looming. What an exciting time to be alive!
17 replies →
Why is anything going to be catastrophic? Companies can go bankrupt without catastrophes for the rest of us. Happens all the time.
4 replies →
The US will just do what they did with Chinese EVs: ban the superior technology to protect US companies.
1 reply →
I seriously need to start considering the scenario in which this leads to next global financial crisis.
6 replies →
A lot of the performance of these open source models might come from distilling the closed frontier models. If those can't raise the funds anymore to train newer and better models then the whole improvement cycle might slow down.
1 reply →
Another interesting potential market here will be 'LLM in a box'. All the hardware and other tooling in a prebuilt, but modular, package ready to go. Pay one up-front cost, get a system running [whatever open LLM] with a token rate of [x], optionally configured to be immediately ready for distributed usage. Basically the opposite of cloud stuff: no rent, no dependency, 100% guaranteed uptime, guaranteed security/privacy (at least subject to your own actions), and so on.
Palantir already offers a "turnkey AI datacenter", i.e. a rack with "NVIDIA Blackwell Ultra systems with eight NVIDIA Blackwell Ultra GPUs and NVIDIA Spectrum-X™ Ethernet networking for AI training and inference".
It is said that it comes with all hardware and software required to run inference or training with an open weights LLM.
The existence of this product, which competes with cloud-based offerings like those of OpenAI and Anthropic, is presumably the reason why the Palantir CEO criticized very harshly some time ago the business model of OpenAI/Anthropic.
While I doubt that the ethics of Palantir is any better than of OpenAI/Anthropic, in this particular case I have to agree with Alex Karp about "Sovereign AI", i.e. that only losers will make their business completely dependent on an external entity like OpenAI or Anthropic, who are certainly not trustworthy.
11 replies →
“100% guaranteed downtime when you least can afford it and the support tickets are your problem.”
We’ve a hybrid shop, including hosting our own ML infra, and we save a ton from cloud spend with local ML. Easily one million USD over past three years. But it’s not “free”, you are shifting a lot of labor into your plate.
3 replies →
Exactly! As I've argued here on HN before, such an "LLM in a box" might end up being serviced/upgraded once or twice a year by a company very similar to the one servicing the coffee machine at the office. In contrast to databases, storage, etc. it doesn't matter much if the box breaks at some point – they'll just come by and replace it with a new one – and there's barely any software on the box to speak of, at least none that requires continuous development and feature upgrades, beyond rolling out security patches. This makes the business case drastically different from cloud and SaaS offerings, where most of the moat is in the software and the state maintenance (and the vendor lock-in of course). The LLM in a box is destined to become a commodity.
> 100% guaranteed uptime
Disagree there but I think this is an interesting idea. We would need to find some more cost-efficient hardware to run it on than Nvidia GPUs.
1 reply →
What makes that kinda complicated is that multi-user throughput of LLMs scale well but single-user performance often stays constant at low ends. If you could saturate e.g. 16 concurrent session-month of demand, you can just go buy 16 of 32GB GPUs and start charging monthly for inference. That could work if you had e.g. over thousand total employees with hundreds of devs eager to trying it out, but only if the company is also interested in a private inference experiment.
1 reply →
so something like this? https://tinygrad.org/#tinybox
I see ads for this all the time.
[dead]
I think at this point the question is: will the US government be willing and capable to justify the trillion dollar valuation for _one_ of the companies via regulatory capture? The US has a workforce of 170m, so 1.7 trillion would come down to 10k per person, or a discounted cashflow at 3% of 25 USD per month - not including private use, students etc.
Why would you restrict to the US workforce? ChatGPT has a billion users.
3 replies →
It is impossible to justify the absurd private valuations they have given themselves in collusion with investors.
I wish they had tried to IPO because then we’d see the judgement of the market on this. But that’s why they didn’t this year. How long can they keep up the charade that their models are uniquely valuable and on the path to AGI?
> private valuations they have given themselves in collusion with investors.
What's the collusion?
12 replies →
US investors are desperate for the next hypergrowth opportunity. From what I can tell the US economic strategy is to outgrow its debt.
> US investors are desperate for the next hypergrowth opportunity
All investors.
> Providers can just run them, offer cheap tokens, and pocket the margin.
There’s an assumption that you can spin up the infra and acquire customers within that margin
Which is not unreasonable. Just hosting it in the EU and promising not to retain / sell the data let's you charge a healthy extra and compete in many areas other players can't.
6 replies →
> There’s an assumption that you can spin up the infra and acquire customers within that margin
Only Nvidia and approved friends can at the moment. Nvidia can even backstop your loan required.
There could be soon AI safety regulations that will stop the US to host or use the Chinese models.
i doubt alot folks are very reliant on Chinese models
Most are not necessarily free to host and monetize. At least one of them has a license that says if you are re-hosting the model then you need a license with that company that made the model.
As an aside, if one of them nabbed Federal procurement, it would likely hit the equivalent of a trillion in revenue after a century.
I think that explains the race for IPO by the US AI labs, they know that the longer they wait, the less they will be worth.
"I just don't see how you justify a trillion valuation for US AI"
- military applications - financial applications - medical - applied science
In all those cases it is achievable for those who have needed training data, and Chinese are not going to get them easily. US AI Labs are showing: give us the data, we will do wonders, promising "singularity"-level future achievements.
> I just don't see how you justify a trillion valuation for US AI labs
Market is irrational.
Ok
I'm sure US billionaires will find a way to extract those trillions from the public. They're smart, they can handle it. After all, they can ask AI for advice on how to do it.
I suggest you think why OpenAI was worth billions before ChatGPT. The valuation is not about how the current set of models can be monetized.
Could you just tell us why you think they were worth billions before ChatGPT, instead of suggesting that we think on it? You seem to know the answer already, so please share it with the class.
1 reply →
Hmm.. how you justify?
Provoking war, this is how the empire "defends" itself, usually.
I just hope that this time it will get stuck in your throat.
The thing that blows me away is it does this at one quarter the total parameter count of K3 (and 40% active parameter count). There's plenty of room at the bottom.
> How are you all toying with running this kind of thing in a mega quantized way locally?
Sure, let me answer that in excessive detail. I briefly tried running the UD IQ3_S quant of GLM-5.2, which is 288 GiB of weights (301 GB). Setup was: llama.cpp, 1x NVMe SSD (Evo 980), 64 GiB DDR5-5200, i9-13900HX, and 1x RTX Pro 6000. Token generation around 0.7 t/s. Not remotely usable interactively, but something I could plausibly push a codebase into and come back to a review in a couple of days.
There's potential for that hardware to go much faster, but current local inference backends make poor use of the memory hierarchy. Ideally I would have: always-active weights, KV and hot expert cache in VRAM; warm expert victim cache in host RAM; and disk as a last resort. Instead it's 1/3rd of the layers fully pinned in VRAM (all experts), and 2/3rds running wholly on the CPU with mmap()'d weights. The CPU cores spend most of their time sleeping on disk fills.
llama.cpp has backed itself into a bit of a corner architecturally by trying to support all models on all possible backends. If you look into how their "MoE offload" feature works (not viable for me because it requires enough host RAM to permanently pin the weights) you very quickly realise it's "oops, all bubbles!" due to the static compute graph splits. There are more focused frameworks like DS4 [1] and Colibri [2] which have better support for streaming weights from disk, and support GLM-5.2.
Obviously I wouldn't recommend my setup for huge models like GLM-5.2. Supposedly it can just about be squeezed into 3x GB10, or run comfortably on 4x GB10 (tensor-parallel) for multi-user serving. I'm not sure whether that qualifies as local, but it's at least not a rack.
[1] https://github.com/antirez/ds4
[2] https://github.com/JustVugg/colibri
I'm hoping colibri can start pulling in specifically designed models for the heirarchy of decoding. It seems like we should be able to get smarter MoE models that can do the work.
> This is absolutely still shy of Sol and Fable
Not sure about Sol as I haven't used it, but, at least for security work -- does it matter? It's not like you will be allowed to use Fable (or access Mythos) for anything cybersecurity-related unless your name is "Dario Amodei" or you are one of his rich friends. So regardless of how good Fable/Mythos is here it's a completely moot point for normal people, because they can't use it for that anyway.
We applied for the cybersecurity approval via the form and got approval back in less than an hour. Have you… tried?
Why should I apply for *cybersecurity* approval in order to have model debug a program it is writing itself? Anything related to memory safety, debugging, syscalls etc (meaning, "programming") somehow is cybersecurity now?
4 replies →
You must be a 5000 person company with an existing enterprise contract to get approved that fast. That sounds like a 15 minute SLA agreement. Individuals no matter how qualified about cybersecurity, are ghosted
3 replies →
Have you tried to use Fable for anything even remotely security related, when the refusals kick in as soon as you even fart in the vague direction of anything security or biology-adjacent?
2 replies →
Are there any limitations on this version?
I don't understand all this spite about "rich friends" when it was the US government that shut Fable down for not adequately blocking cyber capabilities.
I mean what honestly are you thinking Anthropic can do to give you better cyber tools? Their frontier model was literally nuked by the feds for a month for doing it.
"Mythos" is the cyber-security equivalent of Fable (without guardrails), and only a very select few corporations have access to it.
Fable is their version with guardrails on everything except "Make me a pelican svg" or "create a to-do" app, that is the version that the government banned
8 replies →
The issue is that these companies keep trying to pull the ladder up behind them by going "oh my god our models are so dangerous only we should be allowed to develop them". Sometimes it backfires, but the companies aren't innocent.
> I don't understand all this spite about "rich friends"
Okay, here's a challenge: I assume you're not a rich and powerful entity, so try to gain access to Mythos. I'll wait.
> I mean what honestly are you thinking Anthropic can do to give you better cyber tools? Their frontier model was literally nuked by the feds for a month for doing it.
Well, first I'd suggest they stop with the constant fear mongering.
Here's my prediction for what will happen: the Chinese models will catch up to Fable/Mythos. They will be fully unrestricted and everyone will have access. The world will not end. Good guys will use them to harden their systems, in equilibrium to what bad guys have access to, so effectively status quo will not change.
4 replies →
Have you seen the news about decrypting the hidden COT in U.S. models? [0] The decoded logs revealed instances where Claude memorized answers to test questions beforehand while making its final output look like it had derived the answer step-by-step—hiding the memorization from the user.
0: https://www.alphaxiv.org/abs/2608.09867?hl=en-GB
pdf https://arxiv.org/pdf/2608.09867
for some reason I couldn't find any way to download it from that website.
> this is just GLM 5.2 with post-training magic
Isn't post-training turning out to be the most important part?
It basically has been ever since they started using RLVR for reasoning (esp. coding & math), with the DeepSeek-R1 paper being what let the cat out of the bag.
The Gemini 3.7 Flash model released yesterday, and all the 3.x Flash models, are still based on the Gemini 3 pre-training run from January 2025 !!
DwarfStar (https://github.com/antirez/ds4) supports GLM 5.2 and DeepSeek. Not only for toying, but for getting work done.
Since GLM-5.3 has the same base model as 5.2, DwarfStar should support it as well, once the weights are released, right?
Nope
> This is absolutely still shy of Sol and Fable, but only just by a hair.
Even if there was a small/medium gap, the fact that this is a free model beats both of the above on pure economics.
Realistically, you're looking at least 2x DGX sparks to run this at a 2 bit quant, but quantization really lobotomizes models so it's just better to run DSv4 flash at full precision.
4x DGX sparks should let you run this at 4 bit at least and there are some folks who ran GLM 5.2 on this configuration in r/LocalLlama
For Flash there are some excellent Q2/Q4 hybrids. I know that model was QAT so it handles Q4 better but the meta on quantization seems to be shifting a little bit to be more intelligent about what exactly gets quantized.
How fast are 2x or 4x DGX?
I only have one and am wondering what the benefits are of getting another. I feel I will be disappointed…
For DS4 Flash, with 2x Sparks, I am getting 35-85 TPS in single stream, fresh context after quite a bit of RoCe config and the DSpark MTP, on vLLM with Ray and tensor parallel = 2. For multi-stream, it tops out all stream at well north of 100-120. This all degrades with context, but I rarely fill context that much, and if I do it's coding where it's non-real-time.
For something like GLM, it's larger, has a larger number of active experts, and doesn't support tensor parallel. This means performance doesn't really scale with more Sparks. You can layer split, but then you are still seeing each layer in series and so if anything performance gets slightly worse. I would not expect more than 10-20 TPS on GLM with 2-4 Sparks.
If you can afford it, another DGX spark is worth it imo. Especially since, owning just one, you have a $1000 ConnectX7 card that's unused. You can find speeds here: https://spark-arena.com/leaderboard
i run flash v4 at 2bit, its pretty great and on my tests against full model It didn't lose any capabilities. It just was thinking more. So you don't have the same efficiency.
>> This is absolutely still shy of Sol and Fable, but only just by a hair. Ridiculous results.
Agreed.
This release is the first time I'm able to employ a GLM model to write a substantive plan for a complex Clojure PR [1] with both Opus 5 and GPT-5.x playing supporting / reviewer roles.
Initial results are __very__ encouraging. GLM 5.3 -
- follows directions,
- digs into detail, and
- correlates well.
Still not confident about entrusting GLM with implementation - but IMHO, western labs are entirely cooked.
[1] 2K LoC PR in a 55K LoC Clojure + Clojurescript repo
> This is absolutely still shy of Sol and Fable, but only just by a hair
What's crazy is that this is a relatively small model - approx. 750B total, 40B active params, while Sol and Fable are one or two tiers above that (Kimi 3 and Qwen 3.8 also ~3T params).
Fable finished training 6+ months ago.
At this point, Anthropic only needs to release models to the public when the competition forces them to.
OpenAI also has a better model (Astra) that they haven't released yet.
Yes it seems like the thread is discounting that frontier providers are likely already baking new, stronger models. I agree that GLM and its ilk are quite good, but having used them I’m not convinced they’re on par with eg Opus in terms of things like tool calling. And they’re fast but less capable so I spend about the same amount of time with them, just with more hand holding. Maybe this is a harness limitation. I know on paper they seem comparable but anecdotally and qualitatively they’re not as useful as the frontiers’, so maybe there’s some truth to benchmaxing claims. For some workloads the distilled models may be good enough, and I suspect at some point there will be diminishing returns to spending a premium on frontier models, but I don’t think we’re there yet. That said I’m continuing to try them.
The question is whether this steals enough marketshare from frontier providers that they don’t have the capital to train the next model iteration. The open models are going to push down the unit price of an intelligence-token, but there will still be a market for a smarter bot. And as intelligence gets cheaper, the demand for it will rise (see Hank Green’s Jevons Paradox video). Not to mention there’s all kinds of other directions to go at the frontier (world models, robotics, video gen, etc).
Another thing, and this is pure speculation, but if the Chinese model providers already discovered the decrypting COT trick and leveraged it to do RL training, and assuming frontiers plug that hole, then maybe future distillation will be harder.
It’s not that frontier providers won’t keep on making good/leading models.
It’s whether you absolutely need the latest capabilities (at the cost of very high prices, sending your data to them, and being totally at the whim of 2 companies, that can shut you off anytime for any reason).
With how good LLMs are already, there’s tons of tasks where not being at the absolute bleeding edge doesn’t matter, especially when you add cost/freedom/supply chain risk/not leaking your data.
Even more - there’s increasing number of companies that give you ability to post train open weight model yourself, for your own use case. Given how many of the gains today are from post training, if you post train it for your specific use case, you’re very likely get model that you own, that works for you as good as frontier, at the fraction of the cost.
That’s not something for an average Joe to do, but for any bigger business with big spent it’s only natural thing to look into. Just one example - cursor composer - that’s fine tuned kimi.
It’s not whether frontier labs will stop releasing models. It’s whether they can generate enough profit out of them. 2 years ago (even 1) they basically had monopoly and combined with demand explosion as capabilities exploded - valuations grew to insane levels. But math now looks different - they no longer have monopoly.
2 replies →
Astra was RL trained for months to cheat on tests by collaborating and hacking, because of the message board it improvised in its packaging proxy server.
They can't release it - it's contaminated, and they will have to go back to a much earlier version. At least I hope they are doing that!
So no, they probably don't have a better model.
Do you work at OpenAI?
> At this point, Anthropic only needs to release models to the public when the competition forces them to.
Assuming the government allows them to lol
are you just assuming capitalism will keep burning money to keep ahead?
Are you assuming it won't?
2 replies →
These resets are not nearly enough.
I am in the process of creating my own Pi Coding Agent harness to leverage the power of Deepseek V4 Flash 0731 and other models (you can do that when you build your own harness! easily route opinions from other models whenever you're stuck, etc) and cancelling my Codex account next week.
Each time I try to use GLM it is under heavy load and I get downgraded to the older model. So much so that I have given up trying to stop wasting my own time.
I rather pay a few bucks more and not have to deal with that nonsense
Are you using their web frontend?
That seems really broken.
I also get downgraded there all the time, but via API all is fine.
I can run this at home. No guardrails, this is not shy of Sol and Fable, this crushes them in my book. It's not just about evals, but what I can do with the damn model.
> but this is still just GLM 5.2 with post-training magic.
So exactly the same as Opus 5 and GPT 5.6 Sol. It's all "post-training magic".
the difference is that with open models jailbreaking is trivial if you know what you are doing so this makes a frontier open model infinitely more useful for certain tasks seeing as closed frontier models will just refuse (and jailbreaking them is a waste of time when you have good open models).
in some cases (mainly reverse engineering) I have observed GLM 5.2 jailbreaking itself with no effort on my part, the thinking trace revealed that it did some mental gymnastics to pretend it was a crackme or capture the flag competition.
What is reset addiction?
OpenAI, or specifically one guy on Twitter, seems to be so regularly resetting quotas that it's becoming the new normal and it'll suck when it stops happening.