Comment by SwellJoe
13 hours ago
I tried Kimi K3 on a task I've done with every other model I use regularly (https://swelljoe.com/post/i-let-every-agent-implement-its-ow...) and found it chewed a lot longer on the problem and ate up almost the entirety of a 5 hour usage limit on their $19 plan.
I only have the $20 plan from OpenAI and the same task, with a lot of the same implementation details as Kimi Code, only took a few minutes and consumed almost none of the 5 hour limit.
Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare, but when I sat down to add Kimi Code to flar, it was because I wanted to try it on some real work and then couldn't do any, because usage was nearly gone after the trivial task...no other ~$20 subscription I have has felt that tight before.
So, it was really slow to complete the task and seemingly much more expensive than every other model I'd tried. Maybe bad luck. Maybe it'll do better on other tasks. I wouldn't know as I was out of usage when I had time to try.
It did find a bug that Gemini 3.5 Flash introduced unprompted, though, so it has that going for it.
We really need to stop using $/M tokens as the pricing benchmark. I've found that the number of tokens used tends to be a bigger factor than the listed per token price. The cost per task vs. intelligence curve is really what you care about, and in my estimation Chinese models are just not there. They are focused on benchmaxing and getting the highest raw score they can, rather than efficiency.
The artificialanalysis cost per task chart has DeepSeek as the clear winner and Fable as the clear loser. But I would still pick Fable for some tasks, so that also can't be all there is to it.
But I agree that price per token figure is not great. It seems even the tokens per character can vary between models, so it's basically useless.
Yes, this is already accounted for in many benchmarks, but without deep context of the problem type, the top line pricing is the best starting point.
In my own experience, Fable is more token efficient than opus 4.8 with a higher likelihood of completing tasks correctly or at least with minimal corrective work. Opus regularly struggled to gather the correct context and reason effectively about what it had gathered.
GPT-5.6-sol crushes fable in speed and token efficiency and is clearly superior across many tasks that matter for me.
I also find all models from anthropic after opus 4.6 to suffer from the same ai slop language that long plagued OpenAI and seems to have been reduced drastically in 5.6
Don't forget that you are not really seeing the thinking tokens used - so non-trivial to count them.
In my experience, Kimi just tends to think a lot, with the main thing that takes up a lot of space is it constantly second-guessing itself. I've watched it do paragraph after paragraph of "Wait, actually..." while it stumbled and used a ton of tokens on one small detail of what it was asked to do. Though I also gave GLM 5.2 a task to port some JS code to Python to test it, and in my experience it doesn't second guess as bad as Kimi does, but it really did there. It kept doing web searches and second guessing tons of tiny little things, using $0.25 of API spend in total to port about ~50 lines of JavaScript. It did produce an error the first run, but on second run it gave me a program that ran.
I gave Claude Code/Fable the same task and it took significantly less time, but also stumbled on the same error as GLM. I didn't have it fix it though. I was mostly interested in timing differences.
I do like open models where I can, but I'm really hoping they get trained to second guess less. Or maybe I just need to prompt them differently. I'm not sure.
Earlier today I made Claude code implement a feature with fable. It worked roughly 60 minutes and used around 30% of my 100€ subs 5h sessions.
Then I typed /code-review in a second terminal/clean session after the analysis was done (no code changes) the usage was 99%. I then asked it to write that into a review.md so I could restart from that the next day. Sadly the last % wasn't enough for that.
Ymmv, these models behave very differently with no discernable reason. Usually reviews(even with fable) take like 10-20%... Yet suddenly you get it to burn through 65-69% in 15 minutes or so
/code-review in Claude Code spawns a lot of sub-agents (counted like 8 once), each looking at the code from some certain aspect (like correctness, maintainability, duplication, testing, etc). It eats tokens like crazy doing that, but also covers quite a lot. The default code review in Codex does far less (feels like it's only correctness) and doesn't uses subagents. Actually I made a skill for Codex that does a review closer to what Claude does by default, but using like 4-5 agents and some being cheaper models/less than xhigh reasoning. I'm getting pretty nice reviews with that that cover more than just correctness.
> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare
AI subscription pricing is so goofy. You get some amount of usage that varies by models, is measured by opaque token usage, driven by how many tokens the (usually) vendor-provided interface (or model itself) wants to use. Then your usage is limited by time opaque time windows.
You call it goofy, in a different context we would call that a dark pattern, shady, prone to fraud
AI subscription pricing was fine when it was $100/month for some opaque 5 hour token budget I don't think I ever used, not even that one day where I coded for 14 hours non-stop using Fable. But like most people with low token usage, I had a human in the loop and and I didn't use workflows with swarms of agents.
Now, of course, the plan is to remove Fable from the subscription. To paraphrase Darth Vader, they have altered the deal. Pray they do not alter it further.
Gpt 5.6 is still like this at least for the $200/month option. It’s also always faster than fabel. Fabel might be able to do some things better but I don’t have time to constantly wait and find out.
They're probably not going to remove Fable. They just extended it for another month.
Yeah I've noted this behavior with best in class open weight models. They said K3 would have token efficiency improvements and I was hoping especially solving the thinking loop issue that plagued K2.x but even if this release helped somewhat, it looks like we still have a long way to go here... I'm not sure what's up here but I suppose lacking finetuning quality.
What OpenAI in particular have done with reasoning efficiency in the past few months since ChatGPT 5.5 is nothing short of remarkable. It's overshadowed a bit by the benchmark game and the Fable hoopla.
Now is the time to focus less on token cost and intelligence, but tokens to solve a particular set of tasks in closed benchmarks for a variety of categories.
What is the use of grand intelligence if it either costs you a kidney or can't complete at all within a token budget? Even if there are niche uses where you truly want "maximum power" above all, we need to at least more severely penalize such models versus those that does it just as fine within a tenth of the token cost.
I'm aware of some benchmarks at the Artificial Intelligence site, but CLEARLY we are not focusing enough on these today and still leaving the fun surprises to the users.
Yeah, I'm finding I end up switching to Codex and GPT 5.6 a lot lately because I've either run out of Fable usage or Fable refused to do the task. Most recently it refused to work on a WiFi configuration UI for a robot. No idea why it thought that was related to security, biology, or some other sensitive topic. They've hobbled it with guardrails that are overzealous and now there's a big opening in the market. Fable may be the best, but if it won't do the job half the time, it stops being my go to model as I don't want to waste time only to find it refuses halfway through.
Kimi K3 only supports "max" reasoning effort right now, but they plan to enable other levels soon [1].
When I looked at traces from benchmarking, I saw a lot of backtracking and uncertainty while reasoning ("wait, but..."). This also happens with GPT 5.6 and Fable with xhigh/max thinking, albeit to a lesser degree.
I think that explains part of the token inefficiency. Hopefully it will improve with lower reasoning effort settings.
[1]: https://platform.kimi.ai/docs/guide/use-thinking-effort
Absolutely do not pay for the kimi plans thinking they will be cheaper. If you sign up with a Chinese phone number, you can get the same plan for 200 yuan instead of 200 usd, it also only accepts Chinese payment methods iirc. So the plans are really made for Chinese userbase.
That sounds complicated. I'll just use my month of Kimi and then cancel. I have too many AI subscriptions to use them all, anyway. I subscribed mostly to test it. I mean, if it turned out to be competitive, I would keep it, but if it doesn't turn out to really excel and anything and also take longer than Claude or OpenAI models, I'll stick with them.
It is complicated, but paying for the cheaper usd plans really don't get you much usage.
Wow! Does it accept a foreign alipay/wechat pay account?
Those exist?
YMMV, but I'm a US citizen with a KYC'd Alipay account and I was able to subscribe for the Chinese price...
... but I borrowed a friend's +86 phone number, which you'll need to even see that price. or maybe a 回国 VPN will work.
No idea lol, didn't even know those exist..
how do we get chinese phone numbers?
esim
2 replies →
Is Kimi K3 subsidized as hard as the other models out there?
From the end of the month it will be served profitably by other providers around the world, like Kimi K2.6
Sure, but at those API rates it doesn't beat the subscription plans for OpenAI.
Not sure how the economics work for the Chinese models, but DeepSeek did the same task for a dime.
In my opinion, for the vast majority of use cases, DeepSeek is still the most cost-effective model by a mile. $10 feels like it lasts forever.
3 replies →
Does it matter? As an end user I really only care about 1) how much I can do in a week, and 2) how long each task takes.
Subsidies would affect 1, but not 2. But if some VC wants to subsidize my Claude or Codex or whatever, awesome.
The more important question than subsidy is what is the tokenomics of running the model. If it's inefficient to run on an nvl72 cluster (or whatever the heck has enough vram to run a 3T parameter model), and k3 isn't very token efficient, then it might not be that compelling of an open weights model.
1 reply →
It doesn't matter if you can switch easily. It might matter if there are barriers to switching.
Subsidization could affect both of those. If you have $200B in the bank you can afford to throw massive compute at every single request; if you are less well funded, you might throttle more aggressively.
Additionally that same VC could be (read: is always) spent on developing the harness, and other infrastructure around the model, not just the model itself.
So it's apples-to-oranges when comparing a relatively new model to established competitors (i.e. OpenAI @ $900B funding vs Moonshot/Kimi's $30B FYI) because every new model they release is judged on "performance" which is not strictly speaking derived solely from the model.
It's possible Moonshot could get similar performance over time as the build out the rest of the infrastructure. We have no way of knowing how much of OpenAI/Anthropic's success is due to the model vs intelligent tooling built on top of it.
Do models know when they're being benchmarked? I also used K3 briefly and noticed it spent a very long time thinking and obviously a lot of tokens.
However I've seen some benchmarks say it uses fewer than fable which hasn't been my experience.
It would be really interesting to redo the public benchmarks for kimi k3 but token normalize the costs. Ok so maybe k3 beats fable on terminal bench, but how many tokens did it use?
Aren't they still locking reasoning to "max" pending adjustments to support shorter reasoning levels.
It let's me choose different thinking levels in Kimi Code. Not sure if it actually works, yet, but it says "Thinking set to high." when I change it from max.
Why would they do that? Sounds terrible
> tried Kimi K3 on a task I've done with every other model I use regularly and found it chewed a lot longer on the problem and ate up almost the entirety of a 5 hour usage limit on their $19 plan
ArtificialAnalysis puts Kimi K3 just below DeepSeek v4 & GLM 5.2 in token use per task, which is about 2x to 3x more tokens than Grok 4.5: https://x.com/ArtificialAnlys/status/2077832879187620192 / https://archive.vn/zBbFi 2 other open weights MiMo v2.5 & MiniMax M3 are comparatively thrifty.
> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare
I always put my coding subscriptions (that allow it) through "AI gateways" (Cloudflare & OpenRouter are free) which help track token use.
In my experience, Kimi & Qwen Cloud have opaque & restrictive limits, their "credits" drain faster. I now make it a point of subscribing (directly [0]) with providers that are transparent like MiniMax, DeepSeek, Xiaomi, & Z.ai.
[0] OpenCode Go, Cline, and AtlasCloud have generous limits for open weights, otherwise.
With the obligatory disclaimer that I’m impressed with what open weight models can do, I have the same experience with all of them.
The benchmarks come out and say they’re as good as Opus from N months ago, then I use it for a complex task and it doesn’t work as well as Opus from N months ago did when on similar problems.
There’s a real wow factor when you get an open weights model to do amazing things, but in my experience the gap to the frontier models has always been bigger than the benchmarks would lead me to believe.
There can be a lot of value in having the cheaper open weight models for chewing through lower complexity tasks (non-programming in my primary use case) at a cheaper rate than OpenAI or other frontier API costs. Even with those I can measure bigger gaps to the frontier models than the benchmarks suggest.
If the benchmarks aren’t being directly gamed, there’s at least some selection happening where training data or model structures are being picked in ways to maximize public benchmark performance. All of the labs know there’s immense value in having good benchmarks to show for your model because most LLM consumers are picking based on lab provided benchmark charts, not running their own evals. Running your own evals is hard and expensive.
OpenAI measures token efficiency. Look at the API cost charts in their announcement: https://openai.com/index/gpt-5-6/