Comment by mapontosevenths
17 hours ago
They're doing its because they recently cut plan usage by about 50%, leaving Anthropic providing about 5x the value on their plans. [0]
The "cost per task" metric is just a weaselly way to sound like they didn't just double the cost for most users, and aren't having their proverbial lunch money stolen. They can no longer claim to provide more tokens, more capable models, or more usage in general. So they made up a new metric where "task" could be defined to mean whatever they need it to mean.
They had to become VERY specific in their wording to present any kind of value proposition that will entice people.
[0] https://newsletter.semianalysis.com/p/anthropic-subscription...
> leaving Anthropic providing about 5x the value on their plans
5x is calculated in terms of API costs, nobody pays those extreme Anthropic prices outside enterprise users.
> like they didn't just double the cost for most users
Most users aren't on their $200 plan. Most users (including on the $200 plan) continue being subsidized.
Enterprise users are the majority of the users in value. Subscribers are the loss leader hobbyists.
> 5x is calculated in terms of API costs, nobody pays those extreme Anthropic prices outside enterprise users.
Ignore value then. Anthropic still provides about 3x as many raw tokens. The 20x\200 plans provide about 10B for openai versus almost 30B for Anthropic.
Do you really think that Opus uses 3x as many tokens as Sol per task? I don't.
> Most users aren't on their $200 plan.
Fair enough. I should have said most heavy users.
> Most users (including on the $200 plan) continue being subsidized.
They continue to be fifty percent less subsidized, I guess.
Look, I don't fault them for trying to become profitable. I fault them for the weird weaselly business speak about cost per task. It's BS game playing and we all know it.
I'm honestly rooting for them, and hope it all works out. For now though I'm pretty salty that my bill jumped by 150% in one month. Gaslighting about made up metrics just makes me more angry.
How do you know? Have you done the A/B test yourself?
> Do you really think that Opus uses 3x as many tokens as Sol per task? I don't.
You don’t have to speculate. Pull up artificial analysis, pick either opus or sol and scroll down to total tokens and to total price. Opus (and sonnet) use 4x the tokens and 7x the dollars.
That difference alone removes the difference that semi analysis found.
1 reply →
People in HN live in a bubble. Plenty of real world use cannot use subscription plans (not even talking about enterprise agentic use) and is absolutely at the mercy of the cost per task metric.
We agonize over cost per task to the extreme for services/functionality built atop of APIs and can absolutely guarantee you that it is the real world metric that counts most right after “is the model good enough for the task at hand,” and furthermore that, yes, in fact, it is quite often that we see certain models/providers easily using 3x or more tokens on average for the same tasks (or even the exact same task) than a supposedly similarly matched model from another provider. Once you become intimately familiar with the thinking traces from specific models, you can preempt this to some extent with manipulations to the system prompt (essentially heading off entire thinking branches) but the nuking of reasoning traces in the name of preventing against distillation attacks has made this harder.
Cost per task, time per task, quality (good luck measuring objectively between models) are the metric that matter. I don't care how much a token costs, I'm not buying tokens, I'm buying solutions.
Exactly! I called that the Operations Equation and wrote a tiny bit more here about it: https://zero2data.substack.com/p/the-operations-equation
Once you are inserting models into non-creative (and software development is creative work) it is imperative to start creating your own small evals. Not just to check how different models perform in that equation, but to catch model drift over time.
the point is that "task" is a dubiously defined unit that the frontier labs adjust to whatever makes them look good.
Are OpenAI's models delivering 50% (or more) completed tasks than they were last month? Are they delivering 50% more than Anthropic per token?
Nope to both. Not even close.
The only way it works is if you define "task" in a way that OpenAI is good at it and Anthropic isn't.
First of all, only the $200 plan got a cut.
Second of all, barring more cuts, yes I do think the $20 and $100 plans do deliver more tasks because the models have gotten cheaper in the meanwhile (like this very announcement).
> Are they delivering 50% more than Anthropic per token?
So say the public benchmarks. Pull up artificial analysis, “Cost (USD) to run all evaluations in the Artificial Analysis Intelligence Index”:
Sol 6.1 (max): $1,082 Opus 5.5 (max): $8,708 Sonnet 5.5 (max): $7,260
>7x difference in cost for the same tasks.
2 replies →