Comment by Systemerror7A69
4 hours ago
Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?
I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece.
I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentence prompt.
So, it seems to me this "oneshot from a simple prompt" eval is fairly meaningless when it comes to model evaluation itself, as it is in no way representative of real world application.
This would be more something for "vibe coders", people with little to no programming background wanting a website?
I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.
It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.
That's probably because the actual coding benchmarks were saturated several years ago.
As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities.
I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it?
To your point on development work (the code itself), I was talking to some friends on the Google Chrome team about any research understanding the model's preferences around framework ergonomics and abilities to properly implement core web standards for given tasks. I think that would be super fascinating.
I'd assume it comes from Tailwind boilerplate/template sites as it seemed almost all of them were purple at the time
Definitely. It was already a trend before LLMs exploded. And the whole "all websites look the same" has been a thing since at least Bootstrap times.
Purple is just colors #101. ;)
Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like almost who cares what it produces? No one is providing value if the extent of their contribution is so small.
Much more interesting is providing a million tokens of meaningful input and getting 1000 tokens out (high level critique of a detailed design doc, finding a subtle bug in a big codebase, etc).
The eval world is split into:
1. Long form task based examinations like this that test the ability of the model+harness to remain on task, tool calling, overall effectiveness and taste.
2. More direct 1:1 and qualitative comparisons that you might get with a tool like https://evvl.ai/ - which also uses OpenRouter and does similar one off model comparisons (or lets you use it as a MCP from your dev env to be like: "take the prompt from this loop and try it against these other models")
I'm repeatedly noticing that people working at big ai and tech companies are surprisingly not that... good... at using ai? It's like theyre doing a plausible thing to get something done and calling it a day
I've been using code review as my benchmark. Launched a massive parallel Fable/max code review on my lone lisp codebase and recorded all the results and transcripts. Switched to OpenAI and am now running the exact same code review with Sol/max.
It's still a work in progress but preliminary results reveal that Sol is able to reproduce 70%-90% of Fable's performance. This is a very meaningful result for me because code review is what I use AI for.
You touch a point I quickly skimmed in another comment.
Yes, the most valuable benchmarks and evaluations you can write are those that resemble your work.
The evaluations are extremely hard to write and test.
And yes, virtually all benchmarks are E2E one shots, they do not reflect multi turn processes or how most people interact with LLMs.
Which is why every Opus after 4.6 looks better on benchmarks, but is hard to work with interactively.