Comment by ekojs
7 hours ago
> In our evaluations, Kimi K3 delivers frontier-level performance. Among the models tested, its overall intelligence ranks second only to Claude Fable 5 and GPT-5.6 Sol. For the complete benchmark results, see our tech blog. The full model weights of Kimi K3 will be released in the coming days. More details on the architecture, training, and evaluation will be published together with the Kimi K3 technical report.
> K3 pushes the boundary of end-to-end knowledge work. On the GDPval-AA v2 leaderboard, Kimi K3 scores 1687. The benchmark evaluates AI models on real-world tasks across 44 occupations and 9 major industries; Kimi K3 ranks behind only Claude Fable 5 Max and GPT-5.6 Sol Max, and ahead of Claude Opus 4.8 Max at 1600.
> On AA-Briefcase, Kimi K3 scores 1527, ranking second among all models — behind only Claude Fable 5 Max and ahead of GPT-5.6 Sol Max (1495). AA-Briefcase is a private agentic knowledge-work benchmark developed by Artificial Analysis to evaluate frontier agentic capability in long-horizon knowledge work.
Really good benchmark score it seems. Maybe another DeepSeek moment right here.
> its overall intelligence ranks second only to Claude Fable 5 and GPT-5.6 Sol
Pretty sure ranking “second” to two others means ranking third.
Charitably, you could read this as "its overall intelligence [is in a class that] ranks second only to [that of]..."
This is actually what's meant but this bikeshed has been built for yak shaving.
Yeah, bad wording it seems. Though a charitable interpretation is that Fable 5 and GPT 5.6 Sol are joint 1st place in the measurement.
Doesn’t matter, the next one is still third.
1 reply →
If there are two folks standing at gold, nobody gets the silver medal.
3 replies →
While you are technically correct, in English it’s perfectly fine to say it this way as well.
“Second only” here has meaning “next after”, not “number two”.
So... France took second to England and Argentina?
2 replies →
Not if the others tie for first place.
Still third even then.
Which is still great because it means neither of the two best financed labs in the world manage to produce even two models themselves that would beat Kimi K3.
> > K3 pushes the boundary of end-to-end knowledge work. On the GDPval-AA v2 leaderboard, Kimi K3 scores 1687. The benchmark evaluates AI models on real-world tasks across 44 occupations and 9 major industries; Kimi K3 ranks behind only Claude Fable 5 Max and GPT-5.6 Sol Max, and ahead of Claude Opus 4.8 Max at 1600.
This is the same benchmark where Sonnet 5 outperforms Opus 4.8 max.
Like all model releases, the benchmarks aren't going to tell the whole story. All of the open weight models come with amazing benchmark results now. It's hard to believe anything other than that the benchmarks are leaking into (or intentionally included) into training data.
Sonnet 5 does beat Opus 4.8 on several benchmarks. It just costs more and takes longer.
(On several other benchmarks, it costs more, takes longer, and does worse.)
Possible, but pay-as-you-go Hy3 / DeepSeek v4 Pro / MiMo v2.5 Pro (from respective vendors) are genuinely good enough as daily drivers, given the costs (especially, low prices for input cache, which usually makes up 70%+ of total input for agentic workflows). I put in $10 in DeepSeek & Xiaomi MiMo, and I've barely used $1 each, in a week of coding work.
Coding Plans by MiniMax ($20/mo for 1.7b tokens) and Z.ai (~$30/week use for $17/mo) are also tremendous value for money.
i’ll never really understand this comment. why would labs do this if they know private benchmark evals will come out in the next week?
> Maybe another DeepSeek moment right here.
Surely not... What made DeepSeek disruptive was that the cost was 10X lower.
In this case, the cost is about 2X lower the Sol I think?
At 2X, you're pretty close to the error margins due to token efficiency etc...
I'd say this is "on trend" for open models catching up to frontier labs, but its not a "change in the trend" like DeepSeek was IMO.
It was also disruptive because it was open weight, meaning anyone and their dog could theoretically compete with the frontier labs for their inference revenue.
The frontier labs need to recoup a huge amount of cash to cover their model development costs, and justify their valuations. That’s plausible when they’re only ones capable of selling inference on these models, it a lot less plausible when models themselves become cheap commodities, and you’re just competing on your ability to provide compute. Anthropic and OpenAI can’t compete with people like AWS on that front.
cost has nothing to do with why deepseek was disruptive, the fact that it means there is zero moat around anthropic or openai is what's disruptive about it. it means in the mid-term LLMs will be commoditized and customers will flock to the cheapest inference wherever they can find it. there's no reason to stick to the "frontier" labs
It's different, but similar. If they release the weights, then we have a Fable / frontier model people can tinker with. Either way, it's still quite impressive and knocked a US company out of the top three (google). How long before China dominates the top-10 (if they don't already) or the #1 model?
DeepSeek didn’t really change any trends though, unless you count the stock market.
It was impressive work, but models were commoditizing and inference costs were dropping rapidly already. They were neither the first nor the last 10x optimization, from what I’ve seen.
To be fair the stock market is a big one
If you know of any other 10x optimisations currently, please let me know! I'm in the market for a model that's a tenth the price of a frontier model at the same level of quality.
That’s an interesting way to say you’re third. I’m only second to the ten other runners on my local Strava segments.
> In our evaluations, Kimi K3 delivers frontier-level performance
What page does that come from? I'm having trouble tracking it down.
It was on the page linked in the top comment, but it's been removed.
Where are you seeing this write up?
I copied that from https://platform.kimi.ai/docs/guide/kimi-k3-quickstart but it seems they updated the page to remove the benchmark score now.
Where is this from?