Comment by mapontosevenths
15 hours ago
> 5x is calculated in terms of API costs, nobody pays those extreme Anthropic prices outside enterprise users.
Ignore value then. Anthropic still provides about 3x as many raw tokens. The 20x\200 plans provide about 10B for openai versus almost 30B for Anthropic.
Do you really think that Opus uses 3x as many tokens as Sol per task? I don't.
> Most users aren't on their $200 plan.
Fair enough. I should have said most heavy users.
> Most users (including on the $200 plan) continue being subsidized.
They continue to be fifty percent less subsidized, I guess.
Look, I don't fault them for trying to become profitable. I fault them for the weird weaselly business speak about cost per task. It's BS game playing and we all know it.
I'm honestly rooting for them, and hope it all works out. For now though I'm pretty salty that my bill jumped by 150% in one month. Gaslighting about made up metrics just makes me more angry.
How do you know? Have you done the A/B test yourself?
> Do you really think that Opus uses 3x as many tokens as Sol per task? I don't.
You don’t have to speculate. Pull up artificial analysis, pick either opus or sol and scroll down to total tokens and to total price. Opus (and sonnet) use 4x the tokens and 7x the dollars.
That difference alone removes the difference that semi analysis found.
> How do you know? Have you done the A/B test yourself?
Fair point. I just tried it, and I stand corrected. Claude uses almost exactly 3x as many tokens.
In the end Claude produced 2.25x more code, and used 3.01x more tokens (ignoring the cached system prompt, etc). It's answer was also much more substantial and feature rich - including many features I did not ask for.
I gave them both the prompt "Without searching the internet please write and test a python script that accepts a year (in any common format) as input and provides the date that easter fell or will fall on in that year. The output should be in discordian date format. You may not use any libraries."
The thing I found most interesting were the unit tests. Claude implemented the core code using both the Meeus–Jones–Butcher algorithm AND Gauss, but only used Gauss as a sanity check to ensure that it's primary code worked. Sol just validated that it matched some dates it could remember. That explains a good chunk of the almost doubled output, and it was a reasonable thing to do, though neither answer was "wrong."
Token usage looked like this (based on parsing it out of the jsonl, neither provider seems to make it easy to get real billing information per project/conversation.
People in HN live in a bubble. Plenty of real world use cannot use subscription plans (not even talking about enterprise agentic use) and is absolutely at the mercy of the cost per task metric.
We agonize over cost per task to the extreme for services/functionality built atop of APIs and can absolutely guarantee you that it is the real world metric that counts most right after “is the model good enough for the task at hand,” and furthermore that, yes, in fact, it is quite often that we see certain models/providers easily using 3x or more tokens on average for the same tasks (or even the exact same task) than a supposedly similarly matched model from another provider. Once you become intimately familiar with the thinking traces from specific models, you can preempt this to some extent with manipulations to the system prompt (essentially heading off entire thinking branches) but the nuking of reasoning traces in the name of preventing against distillation attacks has made this harder.