Comment by jjcm
5 days ago
> Warning: AI benchmarks. Please don't trust them
I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
That’s true, for example here, this benchmark chooses tasks and datasets that Jeffy is really good at, better than Jev or Laya: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
But the Jeffy models are explicitly trained on those datasets (to be fair, classifiers are just really good, even a decision model finetuned on the same dataset, doesn’t beat a classifier - in classification tasks)
I generally trust benchmarks but my personal experience in the last 6-9 months is they have little relation to real LLM performance, unfortunately.
I've had better luck just running a few of my own simple tests.
It's almost turning into something as good as standardized testing for humans...helpful indicator, but can be gamed.
I like to test the decision models with coin flips. This has a well known answer and I haven't seen one give it back. It undermines my trust in every other decision.
It's a fun question but the models aren't designed for questions like that.
iirc, Jev has a "jaggedness" page that lists known problem areas with the model, and plain mathematics (which I'd argue this is) is one of them.
This is common knowledge though, and not some deep math problem, small children understand this concept.
If they cannot get something so simple right, while at the same time offered as a decision with likelihoods machine, why should I trust their other answers with "probability" they produce?
Here's another post that shows how fragile or inconsistent, even dangerous, these decision models can be: https://zachlandes.com/trolley-problem/
Don't use decision models with important decisions, especially those that involve human lives or irreversible choices
2 replies →
How do you test that? Are you looking for 50% confidence in an answer? I imagine it’s more than that, I’m curious.
A very simple setup, you can try here for Jev: https://jevplayground.com/
It's essentially, "predict the outcome of a coin flip", "what is the result?", and "heads" and "tails" as the answer options. With the Jev website, they do not show the "this is not a real probability" disclaimer until after you get the answer (misleading imo, should be upfront and more prominent, eg. by the input form like the gemini app)
When I give this to an LLM, it will generally respond with something like 50/50, 100% confidence.
They also link to another post they made about AI benchmarks that I really like: https://typesafe.ai/blog/antibenchmaxxing
100% agree, have found through tests a big discrepancy between models optimised for SWE benchmarks and usage in prod - there is a somewhat subjective element to it as well in terms of output that can't be quantified effectively (less so with decision models naturally)
I think the point is we need better and open benchmark players which can bring and monetise on domain specific benchmarks for end users and not just selling them to the big labs.
Or, you know, it's an easy excuse because they're already behind on many of them.
They are? In all my evals at work, Jev seems to win, despite me wanting the open weight versions to.
I imagine if I fine tuned one of the open weight models for my use case it would be possible to beat it, though, but as a general drop in decision model, I’ve not really seen much that beats it especially for the cost
Same here, and I have a project 100% hosted in Cloudflare ecosystem (AI Gateway, Workers, Queues, R2, D1, DO, etc) I was so happy when I saw clef. Tested, and it's bad. Slower and wronger than Jev.
Still waiting on Decisions API using Luna, as this what I was using before.
Run your own evals, folks.
1 reply →
Or they use it as marketing wording to get liked by devs.., which is most likely given we know that’s typesafes usp
we've been public about this before the category was even created
[dead]