Comment by jjcm
3 hours ago
I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests?
Give us something that measures a combination of efficiency and intelligence.
I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have budget to self-test.
Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps?
https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...
Not quite. These cost-per-task benchmarks report the cost of the task after the model gives its initial answer. The total cost is irrelevant, and isn't factored into the model's decisions - a run of the full benchmark for something like Fable might cost $10k.
What I'm looking for is the inverse. I want to give the model a budget of $100, and see how much it can accomplish with that $100. For smaller models, this means they can do more than just choose thinking amount, they can do something like a /loop to keep iterating on a problem until they get it right.
Can something like Deepseek V4 Flash get more answers correct than Fable, when given equal budgets?
Think of it as answering this question: How much intelligence can you get out of a model given a budget of $100? A cost-per-task dash correlates, but it doesn't give you an answer to that question.
It's still spooky to see exponential scales on the money axis.
I do not have exponential funds in my allowance...
“Allowance” has tipped me off / provided a hint perhaps?
If I was younger and had less budget but (presumably) more time, I’d love to be learning about the harnesses and squeezing more out of the open models.
It’s probably generally true that our obligations increase as we get older and the constraints shift around. I’m really enjoying how the frontier models make me more productive, as I figure out how to use them, so have more wiggle room on cost but less time.
Anyway… being nostalgic but I suspect I learned a tonne when cost was the constraint, but was less “productive”.
This is the fundamental question and don’t you find it interesting that there isn’t a nice clean dashboard on the openAI website where we can go and see this metric progress over the release history?
Toby Ord did what he could with public data and it… doesn’t look great.
https://www.tobyord.com/writing/hourly-costs-for-ai-agents