Comment by fiso64

14 hours ago

I think benchmarks like this are too subjective and narrow to be useful. For example, whether a patch "bloats" the codebase really depends on the situation: If it's building a feature that will grow in the future, or refactoring code that has a long history of bugs, then a larger patch might in fact be good. It's not clear from the blog just how much context the LLM judge receives about the long term project goals and history. Benchmarks should be focused on evaluating the final result only. Maybe ask the coder to build a full app, or implement many new large features for an existing app in sequence, with a larger set of requirements, or have another LLM roleplay as the human to make the instructions a little more underspecified. When done, ask a reviewer harness to test the product for 5 hours, not the code. Count the number of bugs and weigh them by severity. "Taste" would then become an automatic consequence of correctness.

(Full disclosure, I'm not a software engineer.)

8 comments

fiso64

Reply
iLoveOncall  13 hours ago

> Full disclosure, I'm not a software engineer

Then maybe you should abstain, because your comment is a complete load of nonsense.

Bad code is bad code regardless of the history or scope of the feature. Maintainability is important because you can never know if a feature will be built upon in the future or not.

Bloat is bad regardless, because it increases the overall complexity of the whole software development lifecycle, for the whole team, forever (or until refactored out): It's harder to keep track of the code and how it works to write new requirements, it's harder to write, it's harder to read and review, it's harder to debug, etc.

You can write extremely poor code that has no bugs, it doesn't make it tasteful. This is simply a ridiculous statement.

  • fiso64  11 hours ago

    >Maintainability is important because you can never know if a feature will be built upon in the future or not.

    Of course maintainability is important. It's almost like saying good code is important (duh). The issue is that what is or isn't maintainable depends on the problem at hand. Sometimes you need to build heavier abstractions or refactor existing code when implementing a feature because it will pay off later. Other times, that exact same approach is horrible over-engineering because a simple, direct fix was all that was needed, so in fact you introduced a maintenance burden. You cannot reliably decide whether a patch is "bloated" or "tasteful" when looking at a diff without knowing where the project is headed.

    >You can write extremely poor code that has no bugs, it doesn't make it tasteful.

    You can, but it becomes increasingly hard to do so as you try to add features and maintain it. Taste, whatever that is, should ultimately lead to a measurable increase in the quality of the final product; if it doesn't, then your definition of "taste" is irrelevant. What I'm proposing is to skip trying to measure this ill-defined concept and only assess the quality of the final product, after the agent spent a significant amount of time working on it, and a reviewer spent a significant amount of time testing it. Agents should be assessed on their ability to build entire projects (e.g., many large features or even an entire app), not just a single feature. If an agent has no taste, then its bad decisions will compound and result in it stalling, or its output having more bugs and performing worse, given a sufficiently large scope.

    • ricardobeat  10 hours ago

      You buy a wooden dinner table, it is fully functional and looks perfect. It’s sturdy. You have dinner on it and it survives a few spills.

      A few months later you find out it is made of PU foam and printed waxed paper. A misplaced knee could bring it down. It’s likely to completely fall apart in a year. Is that irrelevant?

      3 replies →