Comment by CMay
2 days ago
Well, I will say:
#1: it does not require deep world knowledge, because that's not what local models are for.
#2: it directly attacks drive-by understanding, overly linear processing training, poor attention mechanisms, poor reasoning patterns or lazy assumptions that ignore very easy low hanging fruit.
#3: it requires solid instruction following in the face of errors. a lot of models will run into errors and then fall back into some kind of error recovery process that bypasses instruction following.
#4: does not require prompt fine tuning to tweak to each individual model. they all seem to understand.
#5: not unfair. almost every model demonstrates in their reasoning that they have the necessary information that if reasoned about appropriately, could arrive at the correct answer.
#6: not designed to add unnecessary complication that it is intended to exhaust reasoning budgets of any sort, so it is not inherently unfair to models that reason a little more or less. for example, it does not require unnecessary reasoning soaks (ie: hiding the prompt inside base-64 encoding)
#7: has real world use and is probably applicable to overall ability to generalize.
#8: can be scaled up as models get better.
#9: is a very good indicator of how bad a model is falling apart under various inference settings.
I especially like #8. If you have some free time (don't we all have so much of that?) it would be really interesting to run a binary search on each model you have, to see at what size/complexity level it manages to solve the problem, say, 50% of the time.
Are you willing to share this benchmark’s internals? Kinda weird to expect folks to take you at your word without the ability to “trust but verify”
The nature of LLM benchmarking is that they seem to saturate public benchmarks so quick, they are a uniquely efficient case of https://en.wikipedia.org/wiki/Goodhart%27s_law
I'm not asking anyone to take my word, they can believe or not and in practice people should be taking signals from a variety of places and doing their own testing to see how models behave in their own use cases. What I'm measuring and why I'm measuring it may not be the most important metric for your specific use case.
Most other models are simply failing at these tasks. I think the tasks are relevant to overall model capability, but they are not the only metric. You don't give a jellyfish a tool and expect it to produce wonders, so the other capabilities of the model matter.
How much time did you invest in creating this benchmark? Any recommendations/resources you could give on how to do it?